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ABSTRACT 

Papers in this collection result from the work carried out in 
the context of an Invitational Roundtable on System-Level Indicators for 
Higher/Tertiary Education organized by the European Centre for Higher 
Education ( UNESCO-CEPES) and the Research Institute for Higher Education of 
Hiroshima University, Japan. Section 1, "The Roundtable— An Overview " 
contains: (1) "What Was Achieved, under What Conditions, and for What 

Purposes?" (Akiyoshi Yonezawa and Frans Kaiser); and (2) "System-Level 
Indicators for Higher/Tertiary Education: Some Notes on Requirements and 
Use" (pans Kaiser). Part 2, "Instructional Views," contains: (3) "Indicators 

of Higher Education" (Denise Lievesley) ; (4) "On the Uncertain Consequences 

of Taking Advice from Procrustes" (Guy Weave); and (5) "Indicators for 
Tertpry Education Reform: A World Bank Perspective" (Jamil Salmi) . Part 3 
"Npional Views, " contains: (6) "The German Perspective Regarding the Design 
and Use of System-Level Indicators for Higher Education" (Klaus Schnitzer) ; 

( ) Strategic Indicators for Higher Education Systems: Lessons from the 
Frpch Experience" (Thierry Chevaillier) ; (8) "Problems Related to the Design 

and Use of System-Level Indicators for Higher Education in Poland" (Roman Z. 
Morawski); (9) "Romanian Perspectives on the Design and Use of System-Level 
Indicators in Higher Education" (Panaite Nica) ; (10) "A South African 

pppective on System-Level Indicators for Higher Education" (Saleem Badat) ; 
(11) "Key Issues in the Development of Higher/Tertiary Education in 
Cpna (Hong Wei-Meng) ; (12) "The Japanese Perspective on the Design and Use 
of System-Level Indicators for Higher/Tertiary Education" (Akira Arimoto, 
Akiyoshi Yonesawa, Hideto Fukudome, and Masakazu Takeuchi) ; (13) "The 
Ppspectivfe of the United States Regarding System-Level Indicators for Higher 
Education". (Jamie P. Merisotis) ; (14) "Using System Indicators To Stimulate 
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Policy Development" (Herb O'Heron); and (15) "An Australian Perspective on 
System-Level Strategic Performance Indicators for Higher Education" (Martin 
Hayden). Part 4, "Suggestions for the Future and Conclusions," contains: (16) 

"The Past and Future of Quantitative Indicators for Higher Education" (Kalus 
Hufner); and (17) "Suggestions Relative to the Selection of Strategic System- 
Level Indicators To Review the Development of Higher Education" (Richard 
James). Each chapter contains references. (Contains 22 figures and 12 
tables . ) ( SLD) 
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Preface 



This volume represents a principal outcome of the work carried out in the 
context of the Invitational Roundtable on System-Level Indicators for 
Higher /Tertiary Education. This Roundtable was jointly organized by the 
UNESCO European Centre for Higher Education (UNESCO-CEPES) and the 
Research Institute for Higher Education (RIHE) of Hiroshima University. It 
was held from 11 to 13 June, 2001, in Hiroshima, Japan. 

Before going into greater detail about the publication, a few words are 
appropriate with regard to the project itself. The main reasons for launching 
it were directly related to the outcomes of the UNESCO World Conference on 
Higher Education (UNESCO, Paris, 5-9 October 1998), which confirmed the 
need for the renewal and the re-orientation of higher education at system 
and institutional levels. The two basic documents of the meeting - the 
“World Declaration on Higher Education for the Twenty-First Century: 
Vision and Action”, and the “Framework for Priority Action for Change and 
Development of Higher Education”, laid down the conceptual framework, 
the main directions of the reform, and their governing principles. Similar 
documents adopted at regional level served as important inputs for the 
elaboration of these documents. 

Additional objectives of this complex exercise include the undertaking of 
a thorough analysis of currently used statistical indicators, so as to evaluate 
their applicability to present-day higher education as well as to bring about 
eventual modifications or even the formulation of new indicators. All these 
actions are designed to improve the quantitative assessment of higher 
education at system and institutional levels. They were key factors in 
launching the project, “Strategic Indicators for Higher Education in the 
Twenty-First Century”. 

The main responsibility for the process of implementation remains with 
the respective Member States and their higher education institutions. 
UNESCO, in co-operation with other organizations and partners, is expected 
to follow up the developments and to monitor the implementation of the 
vision of higher education as presented in the above-mentioned two 
: documents. Developments in higher/tertiaiy education need to be 
monitored at system as well as at institutional levels, because only by doing 
so is it possible to understand the extent to which specific 
recommendations advocated in the relevant documents of the 1998 World 
Conference on Higher Education are being implemented. 

Providing UNESCO with specific reports on the development and reform 
of higher education in given Member States has proven to be a valid and 
valuable data collection method. It should be continued and applied with 
regard to the World Conference on Higher Education follow-up activities. 
However, in order to be able to reinforce policy and decision-making, there 
is also a need for a quantitative, solid system of factual reporting. 
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Quantitative reporting is difficult, especially at system level, because of 
the diversification of institutional structures as well as the impact of the. 
new information technologies on the provision and organization of studies. 
To represent developments in higher education, indicators almost 
necessanly have to be broad aggregates that are sometimes quite crude. The 
task becomes even more difficult when trying to quantify the qualitative 
aspects which characterize in many ways the basic tasks of higher/tertiaiy 
education - teaching, learning, and research. This task represents a 
particular problem when ranked evaluation is being carried out 
(ranking/ league tables) ^ or when quantitative indicators are constructed for 
institutional and/or programme accreditation. 2 

Collecting quantitative information on higher education systems is not a 
goal in itself. The information provided by indicators will be used to assess 
the state of a higher education system with regard to specific issues. The 
decision concerning the choice of indicators/ issues is to a large extent a 
political decision. For a valid use of the indicators, it is necessary to know 
what the indicator is intended to measure and how the score on one 
indicator may be related to the scores on other indicators. 

There is a risk that indicators may not represent the most important 
issues. They neither reflect the processes nor the operations. 

The full project, “Strategic Indicators for Monitoring Higher Education in 
the Twenty-First Century”, of which this Roundtable was a part, extends 
over the 2001-2003 period. It should be pointed out that its implementation 
was carried out in collaboration and/or with the financial support of the 
following organizations: 

- The Ministry of Education, Culture, Sports, Science, and Technology 
(MEXT), Japan; 

- The Research Institute for Higher Education (RIHE), of the University 
of Hiroshima, Hiroshima, Japan; 

- The UNESCO Institute for Statistics, Montreal; 

- The Division of Higher Education, UNESCO, Paris. 

In addition, a number of international organizations, as well as national 
bodies and selected groups of experts have been involved in this project, in 
various ways. Its implementation was made possible thanks to financial 
support provided to UNESCO-CEPES by the Japanese-Funds-in-Trust for 



An Invitational Roundtable on this question, titled Statistical Indicators for the Quality Assessment of 
Higher/Tertiary Education Institutions - Methodologies of Ranking and League Tables, was jointly organized 
by UNESCO-CEPES and the "Leon Kozminski” Academy of Entrepreneurship and Management, between IS- 
IS June 2002, in Warsaw, Poland. Tlie report of the meeting is available on the UNESCO-CEPES Website, at 
<http://www.cepes.ro>. An edited version of the report along with a selection of the papers presented has 
been published in Higher Education in Europe 11 4 (2002). 

The first meeting of the Working Group on Indicators for Institutional and Programme Accreditation in 
Higher/Tertiary Education, titled Approaches to .Standards/Indicators for Institutional and Programme 
Accreditation in Higher /Teiiiary education was organized by UNESCO-CEPES and held on 5-6 April 2002, in 
Bucharest, Romania. It is to be followed by an Invitational Roundtable on Indicators for Institutional and 
Programme Accreditation in Higher/ Tertiary Education, also organized by UNESCO-CEPES, to be held in 
Bucharest, Romania, on 3 - 6 April 2003. 
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the Promotion of International Co-operation and Mutual Understanding. 
The German Academic Exchange Service - DAAD, offered supplementary 
financial assistance. 

My key partner in launching the project, Strategic Indicators for Higher 
Education in the Twenty-First Century, was Isao Mizuta, currently the 
Director for Educational Policy of the Board of Education in Saitama 
Prefecture, Japan, who, at the time, was seconded to UNESCO as an 
Associate Expert and worked with me in the Section for Policy and Reform 
of Higher Education in the Division of Higher Education of UNESCO. After 
the return of Mr. Mizuta to his post in Japan, his successor, Ms. Satoko 
Imamura, became an important collaborator for the implementation of the 
project. I mention both persons here so as to thank them for their 
contributions to the project. 



Jan Sadlak 
Director of UNESCO-CEPES 
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PART ONE 



THE ROUNDTABLE - AN OVERVIEW 



I. What Was Achieved, under What Conditions, 
and for What Purposes? 

AKIYOSHI YONEZAWA and FRANS KAISER 



1. INTRODUCTION 

Under the rich sunshine just before the humid and hot rainy season, fifty 
participants from fourteen countries gathered on the City Campus of 
Hiroshima University, Japan to participate in the Invitational Roundtable on 
System-Level Indicators for Higher /Tertiary Education. Needless to say, 
Hiroshima is well known as a symbolic city as the result of its sudden and 
tragic “annihilation” on 6 August 1945. Its people respect the work of 
UNESCO in all its meaning. The Ministry of Education of Japan (MEXT), the 
Mayor of the City of Hiroshima, and many Japanese higher education 
researchers, including the staff members of the Research Institute for 
Higher Education (RIHE), welcomed the holding of this important meeting in 
this historic city. 

Mr. Masashi Akiba of the Ministry of Education of Japan delivered a 
keynote address on the current status of the Japanese higher education 
system by making reference to various indicators. The drastic changes in 
the higher education system of the country and the rapid growth in data 
accumulation greatly impressed the participants. For some countries, the 
development of data is not a problem. Rather, it is one of how to deal with 
the rapidly expanding amount of data on higher education. This problem 
represents a sort of “privileged” circumstance not typical of the majority of 
the member states of UNESCO. 

The main purpose of the Roundtable was to discuss the possible 
development of strategic system-level indicators, so that the UNESCO 
member states involved could follow up the developments and monitor the 
implementation of the vision of higher education presented at the World 
Conference on Higher Education - Higher Education in the Twenty-First 
Century: Vision and Action (UNESCO, Paris, 5-9 October 1998). The two 
main documents of the meeting (UNESCO, 1998b and 1998a), respectively. 
Higher Education for the Twenty-First Century: Vision and Action (the World 
Declaration on Higher Education) and the Ftarnework for Priority Action for 
Change and Development of Higher Education (the Priority Action Plan) had 
been distributed as the conceptual framework, the main directions of the 
reform, and their governing principles. 

This overview, consists of four parts. The first part is a summary of the 
presentations given. The second part comprises an expose on the need for 
contextualization. The third part is a reflection on the discussions that took 
place about the requirements and the approaches used in choosing 
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indicators. The final part is a conclusion about where the Round Table has 
brought us. 

2. SUMMARY OF THE CONTRIBUTIONS 

2. 1 . Building the Framework: An International View 

Prior to the holding of this Roundtable, the participants had already 
benefited from a comprehensive work by Abercromby and Fielden, of the 
Commonwealth Higher Education Management Service (CHEMIS, 2000), on 
UNESCO higher education indicators. The two authors implemented the 
detailed breakdown of the two basic documents mentioned above and 
drafted the list of possible indicators related to each goal and purpose. 
Many participants referred to this major contribution throughout the 
sessions. 

During the iirst session, Frans Kaiser presented the framework of the 
Roundtable: what was intended and what should be discussed and, 
hopefully, concluded. The matter was seriously considered. It became clear 
that the task faced by the participants would not be simple, given the very 
complex context of higher education s}'^stems throughout the world. 

Denise Lievesley participated in a video presentation and suggested that 
building systems of comparable data would be an important task of the 
Hiroshima Roundtable. After the discussion of development goals, the 
burden of data collection was mentioned. However Lievesley, rather than 
being pessimistic, argued that the use of priority indicators reduces the 
burden of data collection, and that financial, political, and institutional 
support for statistics is generated when they are used extensively and are 
seen to make a difference to the quality of policy decisions. She suggested 
that indicators should be: (i) realistic; (ii) measurable; (Hi) change-oriented; 
(iu) relevant; (v) universal, and (vi) coherent. According to her, acceptable 
universal indicators in higher education have yet to be developed, and the 
world is changing very rapidly in regard to the diversification of education 
systems. Lievesley discussed extensively the relevance and use of indicators 
in the international context. 

Guy Neave offered a few caveats relative to the increasing reliance on 
indicators, referring to “doxological drift”. In the form that it takes in higher 
education, doxological drift involves a displacement in the function which 
authority imparts to indicators. In theological terms, indicators take on the 
task of enforcing “orthodoxy”, that is, they are viewed as upholding only one 
interpretation of the rich and complex contexts of higher education systems. 
Thus, indicators become stripped of their interpretational ambiguity and, at 
the same time, of their tentative purposes. The caution in regard to 
indicators evinced by Neave is important and should be heeded when 
selecting indicators on higher education. For higher education has a highly 
complex context, and it is impossible to set a simple target. Neave’s 
argument helps to explain one of the reasons why the participants had 
difficulty in proposing a fixed set of indicators, as will be shown below. 
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Based on the rich experience in higher education of the World Bank, 
Jamil Salmi addressed three issues: (i) Dealing with indicators is not only a 
technical activity, but it has an important political dimension, (ii) Indicators 
should not be viewed as threatening. (Hi) There is pressing need to make 
good use of indicators for planning, management, and monitoring. Salmi 
also stressed the importance of asking ourselves what we want to measure 
and for what purpose, questions that can be as meaningful as the answers 
to them. 

2.2. SHARING THE EXPERIENCE: THE VIEWS OF EXPERTS IN DIFFERENT 
NATIONAL CONTEXTS 

Following the contributions offered from an international perspective, 
experts from different national contexts presented their views. Although 
these experts spoke in their personal capacities rather than as 
representatives of their countries or regions, the variations in their views 
were clearly related to the issues they are facing in their respective national 
and regional contexts. 

Western Europe 

Western Europe has a rich context and much experience in the field of 
comparative higher education indicators, based on various international 
experiences shared through the network of EUROSTAT, INES/OECD, etc. 
This setting is very helpful when developing a sophisticated link between 
the national experiences and the development of indicators in a well- 
considered way. 

Klaus Schnitzer attempted to analyze the impact and the relevance of 
indicators given the (political) context. He argued that the political context 
in Germany is not very clear because there are two competing paradigms 
shaping German higher education policy: the old state regulation paradigm 
and the new market regulation paradigm. Both paradigms call for different 
sets of indicators. In addition, the interpretation and use of an indicator 
may vary according to the context or paradigm, as Schnitzer illustrated, by 
citing the “ratio of students with academic family background to age cohort 
of the same background”-indicator. He proposed as a first step the 
. reconciliation of both paradigms by the use of a portfolio presentation in 
which two sets of indicators would be brought together. This approach 
might also prove to be valuable in relating a national set of indicators to a 
set of indicators drawn from the UNESCO World Declaration. 

Thierry Chevaillier argued that indicators at system level are sensitive to 
the structure of higher education systems. He described a number of trends 
and policies in French higher education: democratization (expansion), 
diversification, and professionalization, expansion of continuing education, 
new research funding, and internationalization. All these trends are having 
an impact on the conception and use of indicators at system level. 
Chevaillier’s comprehensive discussion of the various types of indicators: 
financial indicators, input indicators, indicators of efficiency, indicators of 
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quality, and indicators of access and equity is quite useful for 
understanding the current achievement and task in the development of- 
intemational indicators. He argued that some broader existing indicators 
could be useful for more countries but that specific indicators would be 
needed by individual countries to keep track of policies adapted to their 
specific contexts. 

Central and Eastern Europe 

The Central and Eastern European countries also seem to be experiencing 
the processes of building data sets, that, in particular, are compatible with 
those of the Western European countries. 

Roman Morawski presented a comprehensive data list of Polish higher 
education according to indicated tasks (so as to examine the availability and 
comparability of the existing data in Poland). His well-considered analysis of 
the possibilities for applying the data in the UNESCO perspective revealed 
the need to be sensitive to the costs of developing indicators that are 
shareable in the international context. However, it is also true that the 
Polish system is in transition and is facing the rapidly expanding need for 
international compatibility given the role of the system itself in a global 
society. 

Panaite Nica offered an excellent description of the current situation and 
policy strategy of Romanian higher education. Romanians are also 
experiencing the transformation of a national higher education system. New 
laws, the establishment of advisory councils, performance evaluation, 
financing based on institutional agreement, decentralization of decision- 
making, and emphasis on autonomous institutional management appear to 
be the common political agenda that most of the member states are facing. 
Nica’s well-organized consideration of the indications included in the 
UNESCO framework lead to important remarks regarding the future 
development of strategic indicators. His emphasis of the development of 
institutional level indicators was persuasive since he successfully 
demonstrated the structural diversity that each higher education system has. 

Africa 

Participants from three African countries: South Africa, Nigeria, and Senegal 
attended the Roundtable. Hamidou Nacuzou Sail of Senegal made an 
indispensable contribution through his presentation of the theoretical 
framework for the development of higher education systemic indicators. At 
the same time, he illustrated the existing barriers that prevent many 
countries (not only in Africa) from enhancing their capacities to engage in 
effective decision making through an accurate use of well-developed data- 
sets and strategic indicators. 

Saleem Badat of South Africa described the use of indicators for planning 
purposes. Based on the rich experience of South African higher education in 
transition, he argued that the formulation of explicit goals and objectives is 
a necessaiy target of strategic indicators. He also stressed the need for 
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information systems, the capacity to process the information, and the 
importance of generating and interpreting indicators in a clear national 
strategy. 

Peter Okebukola and T. Olaopa made a short presentation that began 
with some basic figures about Nigerian higher education. They raised three 
problems that higher education in Nigeria is facing, namely; fij The Nigerian 
follow up committee of the World Conference on Higher Education is not 
monitoring the implementation of the Action Plan by using the indicators. 
(ii) The accumulation and publication of educational data is politicized; 
therefore, it is difficult to obtain accurate data. (Hi) There is a lack of 
awareness of the importance of strategic indicators. 

East Asia 

The contributions of the two participating countries from East Asia, Japan 
and China, reveal that both of them have been using various indicators 
quite strategically. At the same time, their way of using indicators is 
strongly oriented toward national contexts. In other words, the data 
accumulation was not geared towards the development of (methodologies 
for) indicators in the international context. Of course, their system-level 
indicators do not neglect international comparability: however, in the very 
near future, accelerated globalization as well as the internationalization of 
the economy and of society in this area will most likely push these countries 
to a more internationally-oriented, collaborative way of developing data-sets, 
as has already happened to some extent in the European region. 

Hong-wei Meng’s presentation of the development of higher/tertiaiy 
education in China with a reference to various indicators showed that 
quantitative indicators are utilized in a fairly effective way for monitoring the 
current development of the Chinese higher education system. 

A team from the Research Institute on Higher Education of Hiroshima 
University, Akira Arimoto, Akiyoshi Yonezawa, Hideto Fukudome, and 
Masakazu Takeuchi, traced the historical development of system-level 
information on Japanese higher education. Because of the rather isolated 
development of its system level data set, the Japanese system is facing some 
difficulty in changing its indicators into an internationally comparative set. 
At the same time, its experience of the growing importance of qualitative 
indicators for short-term policy achievement offers a valuable suggestion 
that more attention should be devoted to how “strategy” for “strategic” 
indicators is set. 

North America and Oceania 

North America and Oceania should be categorized as regions that have well- 
developed and diversified higher education systems. This characteristic and 
their orientation towards decentralization and accountability to their 
respective societies have led to well-developed databases at institutional 
level. At the same time, tasks for indicating higher education performance 
are clearly linked to policy frameworks in these countries. These 
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characteristics appear to have led the presenters from these countries to 
make critical but constructive suggestions in order to clarify the UNESCO 
framework so as to “operationalize" it as a set of indicators. 

Through his concise but clear-cut overview of the large multiple 
collection of higher education data in the United States, Jamie Merisotis 
pointed out that the American system is quite decentralized. He argued that 
it lacks integration and coordination, and because of that, it imparts a 
cacophony of information uses and purposes to the data it yields. He also 
mentioned the limited focus on international comparisons in the United 
States. However, Merisotis’s proposal to articulate what higher education 
does for the purposes of policy was clearly based on the rich experience of 
indicating work based on the well-developed institutional-level databases in 
his country. Lastly, Merisotis demonstrated a very interesting conceptual 
framework for reflecting the benefits of higher education. 

Using several indicators in an effective manner. Herb O’Heron argued 
that the different demographic and participation-linked pressures among 
the different provinces of Canada have pushed Federal Government officials 
to give up the idea of “one size fits all” policies. The task of “using system 
indicators to stimulate policy development” leads to the critical conclusion 
that indicators must be used with great caution, for they seldom provide 
evidence of cause and effect relationships. O’Heron also proposed that the 
primary purpose of using strategic indicators should be to help identify 
questions rather than to provide all the answers. 

Reviewing the historical development of the use of performance 
indicators in Australia, Martin Hayden concludes that system-level 
indicators have not been developed to the same extent as institutional 
indicators, which in Australia are highly developed. He argued that certain 
comprehensive statements of purpose and goals at system level may face 
difficulties of measurement. His comment that the global vision of UNESCO 
for higher education may need to be contextualized within Member States 
was valuable. He also proposed that the accumulation of indicators should 
start with a restricted number and scope. The best available model, 
according to him, for such a task is Education at a Glance by OECD. 

2.3. For Future Development 

The last set of presentations contributed to a discussion on future 
developments. 

Klaus Hiifner’s contribution was a reflection on the role of indicators in 
the co-ordination of higher education systems. He argued that in a period of 
transition from centrally planned state systems to more deregulated 
systems in which many higher education systems are present, it would be 
difficult to describe a higher education system through a set of indicators. 
He called for the development of an analytical framework as a starting point 
for the construction of system-level higher education indicators. In addition 
to the development of system-level indicators, he believes that thought 
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should be given to developing regional-level or world-level indicators. Such 
indicators could cover the degree of globalization in higher education. 

The UNESCO Framework for Priority Action ...presents a large number of 
interrelated goals. In his contribution to this volume, Richard James 
attempts to reduce their number and complexity by carving out four 
system-level priority areas on policies and policy-maMng frameworks: ,(i) 
commitment of resources: (ii) levels of participation: (Hi) access and 
retention: and (iv) economic and social outcomes. Based on a substantial 
list of well-documented considerations, James makes a tentative suggestion 
for indicators in these four areas. This framework, he hopes, will stimulate a 
more structured discussion on the selection of strategic indicators (as the 
subsequent discussions prove). 

In the concluding session, Motohisa Kaneko argued that an international 
forum is necessary for furthering discussion and sharing experiences. In his 
view, there is a large gap between existing available data and what is needed 
to serve as system-level indicators. He also referred to the issues of the 
changing behaviour of individual students and the rapid development of 
information technology. 

3. THE NEED FOR CONTEXTUALIZATION 

The discussion that follows indicates the importance of understanding the 
various contexts of system level indicators in the different countries and 
regions of the world. 

3.1. Irfrastructure 

Some countries are faced with a lack or a shortage of infrastructure to 
collect proper information for the development indicators. At the same time, 
the issue of infrastructure is not always limited to the physical shortage of 
resources. Certain social situations such as civil war or a lack of trust 
among higher education institutions and government may lead to a dearth 
of data or to the collection of improper data sets. As Lievesley indicates, we 
should limit core indicators to those that are collectable in most countries. 

Computing facilities and the practical availability of technical support 
staff for the development and maintenance of databases can also be a 
barrier to the collection of required data. In order to develop a data set 
suitable for the national or regional context, the capacity for collecting and 
analyzing necessary data should be built up in the respective countries and 
regions. 

3.2. Clear Goals and Aims 

Whether national governments or international communities have clear 
policy goals and aims, argued Badat, is a critical factor for developing and 
using indicators in a proper way. On the other hand, indicators themselves 
can certainly function as tools for clarifying the objectives and aims of policy 
implementation. As Schnitzer pointed out, indicators may help to clarify the 
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ambiguity of a policy context. The international indicators herein discussed 
should be developed in a similar way, keeping in mind the danger of a . 
global doxological drift. 

3.3. Experiences for Comparison 

Experiences of placing national data in an international context are very 
divers among countries and regions. The Western European countries have 
a long history of doing so. They have developed several international 
databases, such as EUROSTAT and OECD-INES. The Central and Eastern 
European countries have tended to develop their indicators as based on the 

existing de facto “standard” data sets of the international agencies based 
mainly in Europe. 

As Chevaillier suggested, even those countries with a rich experience of 
international comparison are facing a need to develop specific indicators for 
their national contexts. At the same time, some developing countries are 
feeling a crucial need to develop internationally comparable data so as to 
put them on the agendas of the international co-operation projects of 
international organizations and donor agencies. 

3.4. Need for Mutual Respect and Co-operation 

The Round Table discussion revealed the existing and significant diversity of 
physical, social, and cultural contexts among member states concerning the 
development of system level indicators for monitoring. The UNESCO World 
Declaration on Higher Education and the Priority Action Programme cover 
broad visions of member states. Indicating something, however, does not 
mean that standardized goals to achieve are being shared. There is a need 
for mutual respect in a broad and diversified global context. At the same 
time, the sharing of knowledge, skills, and experiences with regard to the 
development of appropriate indicators is the common goal. International co- 
operation for this purpose is, at least in principle, desirable for any and all 
stakeholders. 

4. CHOOSING INDICATORS: REQUIREMENTS AND APPROACHES 

The need for contextualizing, as described above, is a crucial criterion for 
the use of indicators. Particularly when these indicators are used in a 
supra-national context, there is clear danger of oversimplification when 
indicator scores are interpreted. 

In the following pages, the focus shifts from the interpretation of the 
indicators to the process of choosing indicators. The goal of the Roundtable 
was to formulate a concise set of indicators for monitoring the progress of 
national systems towards the situation as described in the UNESCO World 
Declaration and the Priority Action Programme. 

The question of how to go about selecting indicators was raised at the 
very beginning of the Roundtable. Answering that question proved to be 
crucial to its results. 
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First, this study reflects on the requirements for indicators, as listed in 
the Kaiser study. Most of these requirements were addressed in the 
contributions of the participants and the Roundtable discussions. From the 
same contributions and discussions, the participants learned that there 
were considerable differences in the ways in which they tried to achieve the 
goal of the Roundtable. Two approaches were discussed. 

This section concludes with a set of comments on the consequences that 
this debate has had for the results of the Roundtable. 

4. 1 . Requirements Reviewed 

In the article by Kaiser, a list of general requirements for the selection of 
indicators is given. Most of the issues raised by them were addressed during 
the Roundtable. What were the most interesting comments, criticisms, or 
suggestions? 

User-Related Requirements 

Some of the requirements are related to the users of indicators. It may turn 
out that these requirements have different implications if they apply to 
different users. The potential users of the set of indicators identified by the 
participants comprise the international organizations (especially UNESCO), 
the national policy makers, higher education institutions, and the research 
community. All of these users may have a different view as to how the 
requirements listed may affect the choice of indicators. 

An indicator should measure the phenomenon to which it refers (content 
validity). During the Roundtable, some progress was made regarding the 
proposed focus of the indicators. .Some participants were invited to the 
Roundtable from international organizations: others came as national 
experts or from the higher education research community. The proceedings 
proved that participants from these groups had different views of the 
potential use of the indicators the Roundtable was set up to deliver. The 
participants from international organizations and the research community 
had a clear focus on the use of the indicators evoked in the international 
policy context of the UNESCO World Declaration on Higher Education. 
Indicators need to show what progress there is. 

The national experts had a different perspective. Although many of them 
made an effort to relate their national experiences to the World Declaration- 
process, their major focus was on the national policy context. The resulting 
set of indicators should overlap with international indicator sets; the 
international set should cover important national higher education issues 
and developments. Schnitzer cited the need for indicators to be used with 
the new market paradigm in Germany. Chevaillier underlined the growing 
importance of lifelong learning, the ICTs, and the multiplication of 
stakeholders in France. Tadatoshi Akiba and Saleem Bad at cited the 
profound structural reforms occurring in the Japanese and South African 
higher education systems. 
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The content validity of many national indicators as indicators for the 
World Declaration was challenged by a number of participants. Existing 
indicators often fall completely short in representing the World Declaration 
and the Priority Action Programme goals, especially regarding distribution 
characteristics. 

Indicators should be easy to understand for all users [face validity). This 
requirement was not high on the agendas of participants. A few remarks 
were made about the face validity of “soft-quantifiable” indicators (see below). 

Indicators should be relevant to the user. All efforts to develop an 
indicator are vain if the indicator is not used. An indicator will be used only 
if it focuses on a phenomenon that is considered to be important by its 
users. Some of the participants mentioned that the indicators should also 
reflect policy issues that are dominant in their national contexts. Doing so 
would identify the users and their priorities and would therefore be an 
essential step in developing a set of indicators. 

The relevance not only refers to the indicators themselves but also to the 
effort made to develop and implement them. It is no use developing and 
implementing an indicator that has already been developed and 
implemented by somebody else. This point was brought forward in relation 
to the OECD indicator project. In that project (leading to the more-or-less 
annual publication of Education at a Glance), a large number of indicators 
has been developed. The conclusion of the Round Table was that, although 
there was some overlap in the financial/ input indicators, the UNESCO 
project has a much broader scope. 

Indicators should be strategic. The project calls for indicators that reflect 
a process of change and development in higher education systems. 
Indicators should portray change in a phenomenon, possibly related to the 
goals set for that phenomenon. Data on indicators should therefore be 
available for a certain period of time. This requirement was initially brought 
into the discussion as a rather technical criterion (indicators, like indices, 
should be formulated in terms of changes, rather than of steady states and 
absolute levels). 

During the Roundtable, two new interpretations were introduced. The 
first was that of a focus on new developments to be captured. The World 
Declaration covers ground that has not before been touched by existing 
indicator systems. The use of the ICTs, partnerships among and between 
stakeholders, the brain-drain brain-gain phenomenon, lifelong learning, 
relevance to society, and student guidance and counseling are issues that 
need to be developed in the long run, but for which indicators have not yet 
been developed. The second interpretation was a forward-looking 
perspective. When placed in a framework in which indicators are linked to 
each other, they may also be used as a contingency planning instrument, as 
O'Heron explained in his contribution. 
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Other Requirements 

Indicators need to be quantitative. Although the project outline of the Round 
Table called for quantitative indicators, this requirement was challenged by 
most of the participants. It was agreed that the UNESCO World Declaration 
included a number of issues that are hard to quantify. The Fielden study 
already showed that it is impossible to quantify all issues covered in the 
World Declaration and the Priority Action Programme. 

The conclusion reached is that qualitative information, in addition to 
quantitative information, is crucial for monitoring the World Declaration 
process. 

An interesting line of discussion developed regarding the level of 
measurement. The use of nominal-scale indicators,^ or ordinal-scale 
indicators^ was understood by many participants as being a way to capture 
information beyond the scope of existing indicators. In particular, the long- 
term goals of the UNESCO World Declaration and the Priority Action 
Programme, for which policies had not yet been implemented or even 
developed, are difficult to capture with numbers only. Although many 
participants applauded the further development of such soft-quantifiable 
indicators, as Schnitzer called them, some critical remarks were made as 
well. The main criticism referred to problems in interpretation. 
Quantification enhances the presentation options of the information but 
does not always make clear what the numbers mean. 

Especially in an international comparative context, the use of plain 
descriptive information has to be considered as an alternative. 

The data underlying the indicator scores should be reliable. If a 
measurement is repeated, the results should be identical. Either because 
the discussion focused on what to measure, or because the issue was overly 
technical, this requirement did not provoke any significant discussion. 

Indicators should produce iriformation that is up-to-date (timely). In 
practice, this condition seems hard to meet. In a number of contributions, 
problems with timeliness were reported. The time-lag between the event and 
the availability of data on that event is relatively long, which, as Merisotis 
pointed out, may distort the judgments of policymakers when deciding on 
action. 

Collecting data and calculating indicators that meet all requirements listed 
has to be feasible. This requirement refers to the costs of collecting data and 
of calculating indicators, as well as to capacity and time. This criterion 
played an essential role in many contributions. It was the main reason why 
presenters proposed to use existing data and to avoid the deyelopment of 
new indicators. Abercromby and Fielden (2001) mentioned that collecting 
data for all elements of the World Declaration Vision is in principle possible. 



^ Calculations based on nominal-scale indicators cannot be used to generate ranking. An example of 
such an indicator is that of a score of “1” representing a staff development policy that has been adopted and of 
“0" indicating that no such policy has been adopted. 

^ The scores can be ranked, but calculations are not permitted. 
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However, for a substantial part of the Declaration, this task would require 
much time and the collection of new data at relatively high costs. In many 
UNESCO countries, it is not only the cost of developing new indicators and 
of collecting information that are the main barriers to feasibility. As Badat 
pointed out, in many countries the capacity to collect (new) data is very 
limited. 

Indicators should not be used in isolation. Each indicator points at only a 
part of the phenomenon at hand. The combined information may provide a 
true image of the phenomenon. 

The quest for the ultimate indicator, one representing a social 
phenomenon in a single score, was given up a long time ago. The complexity 
of processes like the UNESCO World Declaration on Higher Education 
process calls for a (large) number of indicators, each describing a part of a 
given social phenomenon. The value of such a collection of indicators will 
increase considerably if an effort is made to relate them to each other. 
Drafting and using a framework, even it is not a causal one, is crucial for 
interpreting the scores on the indicators. Using an indicator-score without 
knowing what part of the process it refers to will veiy likely lead to a 
misinterpretation of that particular score. 

A conceptual framework may also help in choosing indicators. In his 
contribution, Merisotis covers a framework to identify possible blind spots 
and overlapping in the list of indicators. His an^ysis is driven by a 
classification of benefits of higher education, either public or private, or 
economic or social. The grid drawn by this classification may Idc used to 
assess the completeness of indicator systems. James uses a framework for 
similar reasons. The input-process output model he uses is condensed from 
the Priority Action Programme. A causal framework may also help to 
identify the crucial indicators for future development, as is proposed by 
O’Heron. 

The need for a conceptual framework was felt by most of the 
participants, but none of the frameworks presented demonstrated in an 
explicit and convincing way how indicators are related to each other. Partly 
this result may be due to the ad-hoc and unsystematic ways in which policy 
makers set the stage for indicator systems. It may be partly due to the lack 
of knowledge of the relations among the various phenomena. 

4.2. Two Approaches 

How to choose indicators? Clearly there is no standard answer to this 
question. Certain classifications as to what influences the choice of 
indicators exist in the literature or educational indicators. Nuttal (1994) 
distinguishes three forces: (i) policy considerations; (ii) scientific/technical 
issues; and (Hi) practical considerations. Policy considerations set the stage 
for the selection process: they determine the rationale for the set of 
indicators. Policy considerations also set a focus on the use of policy 
instruments. Indicators should give a clue as to what instruments may 
work or have worked. 
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Because of this instrumental focus, Nuttal (1994) links policy 
considerations to a modeling approach. For knowing what may work and 
what will not, it is important to have some kind of model in which the 
indicators (or better, the phenomena they point to) are related to each other. 
Although such models are not yet very well developed, they are crucial for 
the interpretation of indicator-scores. 

The second force, scientific/technical considerations, focuses on issues 
of validity and reliability. Is it possible to translate the social phenomena 
that one wishes to study into measurable terms, and can one collect the 
appropriate data for doing so? These questions seem to be straightforward, 
and their answers are crucial for the creation of a certain confidence on the 
part of the users in the indicators. Without such confidence, the indicators 
will not be used for long. 

The third type of consideration is that of practical considerations. 
Feasibility and timeliness are the keywords. It is no use developing a set of 
sophisticated indicators for which data cannot be collected or only at high 
costs. Indicators need to give up-to-date information since most users are 
interested in the current or recent situation. 

In practice, a mix of these types of considerations is used. However, very 
early at the Roundtable, two more or less distinct approaches emerged, 
each with its own mix of considerations. The first approach is called the 
synoptic or conceptual approach. The primary focus is on the phenomena to 
be represented. How can the processes leading to the realization of the 
World Declaration Vision be best measured? There is the strong influence of 
the policy considerations and the .scientific /technical issues. The second 
approach can be called the practical approach. The major concern here is 
feasibility considerations. In a way, this approach starts at the other end: 
from the data available. The central question is how the available data can 
be rearranged and reinterpreted in such a way that they best represent the 
World Declaration process. Among the “followers” of the practical approach 
there was an urge to have some tangible results that they could take home 
with them; a list of indicators they could use in monitoring national 
progress towards the vision reflected by the UNESCO World Declaration. 
Table 1 below indicates in a tentative way the relative weights that the 
; conditions discussed in the section above have in regard to the two 
approaches. 




Table 1. Tentative weighting of selection criteria by basic approach 




Synoptic/Conceptual 


Practical 


Quantitative 






Content validity 


++ 




Face validity 


+ 




Reliability 


+ 


+ 


Set of indicators 


++ 




Timeliness 




++ 


Feasibility 






Strategic 


++ 




Relevance to the user 




+ 



Tg 



30 



A. YONEZAWA and F. KAISER 



As rr^ntioned above, there is no clear-cut distinction between the two 

distinction may help in understanding the 
outcome of the Roundtable. Among the participants, there were partlsmis of 

t . groups were convinced that “their" approach would 

contnbute to the achievement of the goals of the Roundtable. But the 

pproaches do not mix. They are like counteracting forces, keeping each 
Other from achieving a solution. ^ ^ 

5. CONCLUSION 

Roundtable were a heterogeneous group of experts 
the field of higher education and its indicators - heterogeneous in a 
number of ways. They had different affiliations. They ranged from 
international civil servants, to national experts, to members of iie higher 
ucation research community. Many had multiple affiliations. Participants 
vane m the approach they used in choosing indicators, ranging from a 
synoptic and conceptual approach to an approach driven by practical 
considerahons. in addition, all the participants came from different^cultural 
social ^ckgrounds. Their experiences and expertise were formed in 
^^ntext^'' ■ were going to be used in equally different 

u multiple layered heterogeneity, the Roundtable was a success. 

\\r M ?? Pro^^uce a concise set of indicators for monitoring the UNESCO 
World Declara ion on Higher Education process, as had been envisaged in 
toe project outline. It did however touch upon a large number of issues and 
orisiderations that had to be taken into account when selecting a set of 
higher education indicators. The rich discussions may prove to have been 
mulating for those at the national level who have to select and to use the 

taSnt ?■ the Roundtable may feed into the discussions 

^ng place between the various users of the indicators, discussions that 
are essential for their proper and continued use. 
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II. System-Level Indicators for Higher/Tertiary 
Education: Some Notes on Requirements and 
Use 

FRANS KAISER 



1. THE PROJECT 

“There is a need for [a] qualitative, solid system of fact-reporting, even in 
cases of daunting tasks, to quantify intangibles of a set of complex teaching, 
learning, and research phenomena and the administration, functioning, 
and financing of higher education” (UNESCO, 1999). 

The Roundtable on Strategic System-Level Indicators on Higher 
Education that was held in Hiroshima, Japan, from 11 to 15 June 2001, 
should be seen as a step towards such a system of factual reporting. This 
article intends to summarize the main requirements that indicators must 
meet and to present them in the context of the theme of the conference. 

2. INDICATORS 

There is no single authoritative definition of what an indicator is. Johnstone 
(1981, p. 3) described an indicator rather vaguely as “something that 
conveys a general impression with more or less precision - something that 
gives a broad indication of the state of the situation investigated”. 

The discussion as to the nature of an indicator is further complicated by 
the introduction of “performance indicators”. Cave defines performance 
indicator as “a measure - usually in a quantitative form - of an aspect of an 
activity of a higher education institution. The measure may be either ordinal 
or cardinal, absolute or comparative. It thus includes the mechanical 
applications of formulae and can inform, and be derived from, such 
informal and subjective procedures as peer evaluations or reputational 
rankings” (Cave et al, 1997, p. 24). What distinguishes a performance 
indicator from an “ordinary” indicator is that the former is used as an 
assessment of goal-attainment. 

Strategic indicators ai'e another class of indicators. Since the project 
documents did not specify “strategic”, this author used his own 
characterization. Strategic indicators are prospective. They focus on core 
goals set for the system. 

The term, “prospective”, is used to indicate that the strategic indicator 
must inform the user as to where the system may be in the future. How to 
do that? First a strategic indicator should be change-oriented. Indicators 
should identify trends. Using the assumption that trends will continue, 
insights into the possible future situation may be generated. Furthermore, 
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we can use causal chains or flow patterns to “look into the future”. If an 
indicator refers to characteristics that appear early in the causal chain or- 
flow, it may indicate the levels of the characteristics at the end of the causal 
chain or flow in the future. An increase in the number of new entrants in 
year t may indicate that the level of graduates may rise in t + x [x being the 
average time for completion of the programme). 

The second characteristic is the orientation on core goals. In choosing 
strategic indicators, one should ask whether or not this indicator has a high 
impact on reaching the core goals, as set, for example, in the . UNESCO 
World Conference on Higher Education (1998a). 

3. REQUIREMENTS 

In the literature on (educational) indicators, there is a long list of criteria 
that indicators have to meet. These criteria refer not only to the practical or 
technical aspects of indicators. The fact that indicators have to be used 
brings in political (or more generally, normative) considerations regarding 
the choice and presentation of indicators. For the purposes of this 
roundtable, the following criteria were proposed: 

- Quantitative: The project outline calls for quantitative information. 
Thus, the phenomena to be indicated should be quantifiable; 
however, not all the indicators have to be measured as ratios. As Cave 
stated, measurement can also be on interval or ordinal scales. The 
latter scales will be used frequently whenever the “intangibles” 
mentioned in the World Declaration (like cultural aspects) are to be 
quantified. 

- Content validity: The indicator should measure the phenomenon to 
which it refers. Content validity is a crucial requirement. It is also 
difficult to meet, especially reading indicators that refer to broad or 
“intangible” phenomena (as can be found manifold in Higher 
Education in the Twenty-First Century... and the Framework for Priority 
Action.... (UNESCO, 1998b). The fact that indicators are used in 
international comparisons complicates the quest for content validity. 
Indeed, does an indicator mean the same in all countries being 
compared? 

- Face validity: In addition to having a good content validity, an 
indicator should also be easy to understand for all users. The use of 
complicated formulae to calculate an indicator-value is not very 
helpful in obtaining good face -validity. 

- Reliability: The data underlying indicator scores should be reliable. If 
a measurement is repeated, the results should be identical. 

- Sets of uidicators: A cmcial element in the discussion on 
interpretation was that indicators should not be used in isolation. 
Indicators point to different aspects of a certain phenomenon. The 
combined information may provide a true image of the phenomenon. 
In the choice of indicators, one has to make sure that all relevant 
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aspects are covered. To combine the information conveyed by 
indicators, a map of the relations among them is needed. Whether 
this mapping is a causal model or merely a presentation of the stocks 
and flows like the input-process-output model is not of prime interest, 
as long as relations are made explicit. Redundancy within a set of 
indicators should be avoided. If two indicators are strongly related to 
each other, the elimination of one of them should be considered. 

- Timeliness: Indicators should produce information that is up-to-date. 
For many statistics, the time between the event and the availability of 
data on that event is relatively long (three to four years). Such time 
lags may distort the judgment of policy-makers, when deciding on 
action. 

- Feasibility: Collecting data and calculating indicators that meet all 
requirements listed have to be feasible. Feasibility refers to the costs 
of collecting data and of calculating indicators, as well as to capacity 
and time. 

- Strategic: The project calls for indicators that reflect a process of 
change and development of higher education systems. Thus, 
indicators should indicate change in a phenomenon, possibly related 
to the goals set for that phenomenon. Data on indicators should 
therefore be available for a certain period of time. Throughout that 
period, there should be no changes in definitions in data-collection or 
indicator-definition. 

- Relevance to the user: All efforts to develop an indicator are in vain if 
the indicator is not used. An indicator will be used only if it focuses 
on a phenomenon that is considered to be important by its users. 
Identifying the users and their priorities is therefore an essential step 
in developing a set of indicators. 

Not all of the requirements listed above are fully compatible with each 
other. Validity, timeliness, and feasibility may not be maximized all 
together. Trade-offs between requirements have to be made. 

4. INDICATORS FOR WHAT? 

The vision of higher education as formulated in Higher Education in the 
Twenty -First Century comprises eight elements: 

t equity of access: 

tL enhancing the participation and promoting the role of women; 

iiL advancing knowledge through research in science, the arts, and the 
humanities and the dissemination of results: 

iv. a long-term orientation based on relevance; 

V. strengthening co-operation with the world of work in analyzing and 
anticipating societal needs: 

vL diversification for enhanced equity and opportunity: 

viL innovative educational approaches: 

viiL higher education personnel and students as major actors. 
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To achieve this vision, a large number of actions is brought forward. 
These actions refer to: 

- qualitative evaluation; 

- the potential and the challenge of technology; 

- the strengthening of higher education management and financing; 

- the financing of higher education as a public service; 

- the sharing of knowledge and of know-how across borders; 

- the movement from brain-drain to brain -gain; 

- partnership and alliances. 

Most of the issues mentioned above require further systematic use. The 
tasks involved will lead to a proliferation of indicators. Developing a short 
list of indicators that conveys a sense of completeness then becomes a 
formidable task. 

The meeting was focused on the level of the system. In Higher Education 
in the Twenty-First Century (UNESCO, 1998a) and the Framework for 
Priority Action (UNESCO, 1998b) an additional level is mentioned very 
frequently, the institutional level. Limiting the scope of the meeting to the 
system level results led to substantial reductions in the goals or actions to 
be indicated. However, since the system-level and the institutional level are 
related to each other, the institutional level and its indicators should be 
kept in mind when discussing indicators at system level. 

5. CLOSING REMARKS 

By publishing this study on strategic indicators, UNESCO-CEPES is 
contributing to a better knowledge of the subject. Based on this information, 
the policymakers and/or other stakeholders may decide to take action, 
either by making or implementing a policy-instrument or by looking into the 
issue further. However, indicators are an aid to expert judgement, not its 
replacement. 

Although policymakers play a crucial role in higher education policy, 
they are not the only audience of the project. Other stakeholders, like higher 
education institutions, students, and the general public should be informed 
about the progress made regarding the goals set in Higher Education in the 
Twenty-First Century. They must be taken in consideration when presenting 
the indicator scores. 

Owing to the dynamic and complex character of the higher education 
field, indicator results should not be used in a mechanical way. They should 
be used to start discussions with the relevant actors and stakeholders in 
the field. 

The issues raised above do not represent a limiting list of issues and 
problems related to the choices of strategic indicators for the development of 
higher education. Although a great deal of work has been done (Abercromby 
and Eielden, 2000), the challenge to condense the information into a set of 
indicators in such a way that all stakeholders may and will use the set 
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remains a formidable one. The set should be used as a tool to assess how a 

higher education system is doing on its way to the new vision. 
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1. INTRODUCTION 

This study explores how statisticians can exploit their skills and expertise 
effectively to ensure that data collected internationally on higher education 
can assist the development of effective policies in this area. 

International statisticians are faced with dilemmas in their attempts to 
collect data on higher education, wishing to raise consciousness of the role 
of statisticians employed in an international context, to explain some of the 
constraints under which they work, and to address the principles which 
ought to govern their work. 

This study will concentrate on cross-nationally comparable data. Many 
countries of the world have yet to develop nationally relevant systems of 
data collection on higher education. With a view to building systems of 
comparable data, special problems occur in the process of collecting data in 
some of the poorest countries of the world. The context and a framework for 
the collection and interpretation of the data are further presented. 

2. THE INTERNATIONAL CONTEXT 

2. 1 . The Scale of the Development Problem 

Economic and social gains over the last half-century mean that some people 
enjoy a standard of living far superior to that experienced in the past. Yet 
many people remain desperately poor. Half of the population of the world 
still has to manage on less than $2 a day, and 1.2 billion, on less than $1. 
Unless efforts are increased and co-ordinated, the situation will deteriorate. 
The population of the world recently reached 6 billion, having quadrupled 
since the beginning of the Twentieth Century, and is forecast to reach 8 
billion by 2025, with almost all of the additional people living in the very 
poorest countries. Currently, eight out of ten people live in the less 
’ developed countries where fertility rates range from 5.6 children per woman 
in sub-Saharan Africa, through 4.5 in the Arab States and North Africa, to 
2.9 in Latin America and the Caribbean. Some 1 13 million children have no 
access to schooling, and 880 million people, two thirds of them women, are 
illiterate. 

The latest estimates from UNAIDS of the impact of the AIDS pandemic 
are horrilying: at the end of 2000 there were estimated to be 36. 1 million 
people living with HIV/AIDS, of which 5.3 million were new infections over 



* Tliis paper has drawn lieavily on material used by the author in preparing the Presidential Address for 
the Royal Statistical Society (see. RSS, Series A. forthcoming). 
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the previous year. AIDS, a disease predominantly of the poor, deepens and 
spreads poverty. Some 96 percent of the people infected live in the 
developing world, 70 percent in sub-Saharan Africa. Worldwide, it is 
estimated that by the end of 1999 AIDS had orphaned 13.2 million children 
under the age of 15, of whom 12.1 million were living in sub-Saharan 
Africa, compared with 9,000 in Western Europe. Such children are much 
less likely to go to school and are more likely to be malnourished than non- 
orphans. Will they be able to escape poverty? According to the World Health 
Report [WHO, 2000), AIDS caused 20.6 percent of all deaths in Africa in 
1999, twice the percentage of any other single cause. 

These few statistics illustrate the magnitude of the inequalities that the 
world faces. 

Although the extent of this poverty seems to be insurmountable, these 
figures should be viewed against a background of almost unbridled 
expenditure on weapons and consumer inessentials by developed countries. 

- Since 1945, an estimated $8 trillion (USD) has been spent on nuclear 
weapons worldwide, but the estimated public expenditure on 
education represents less than 0.5 percent of this sum. 

- According to some estimates, the world would need to spend an 
additional $7 billion (USD) a year to provide primary education for 
those children not currently in school. This sum is less than the 
Eimount spent annually on cosmetics in the United States or on ice 
cream in Europe. 

- The price of one ballistic submarine ($1,453 million USD) would 
double the education budgets of eighteen poor countries, with 129 
million children to educate. 

2.2. Governance 

What is meant by “governance” when applied to the international realm? 
What are some of its desirable attributes if the aim is to manage 
successfully the transition from an international to a global world? These 
are questions addressed by the Secretary- General of the United Nations in 
his report to the Millennium Assembly (Annan, 2000). He argued that for 
the good of people throughout the world and for the sake of our common 
aims, we must help to strengthen the capacity of individual states to govern 
effectively and help them, in our “fluid, highly dynamic, and extensively 
networked world”, to develop a deeper awareness of their dual role in the 
global world. This duality recognizes that each state not only bears 
responsibilities in relation to its own people, but also collectively shares 
responsibility for the whole planet. The thrust of the Secretary-General’s 
report was that decision-making structures through which governance is 
exercised internationally must reflect the broad realities of our times. 

“Better governance means greater participation coupled with 

accountability. Therefore, the international public domain - including 

the United Nations - must be opened up further to the participation of 
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the many actors whose contributions are essential to managing the 
path of globalization”. 

This conclusion led him to argue that the more integrated global context 
demands a new degree of policy coherence, and he regretted that formal 
institutional arrangements often lacked the scope, speed, and informational 
capacity to keep up with the rapidly changing global agenda. 

2.3. Globalization 

Globalization is often hailed as offering the solution to disparities, and in 
some countries, it has indeed brought about rapid economic growth with 
associated rises in living standards. However, these advances have been 
restricted to relatively few countries and have been distributed unevenly 
within them. 

Globalization is changing the way we live and work with each other, a 
situation that is resulting in increased interdependence. Although 
globalization is a universal phenomenon, its consequences are not the same 
across the world. When viewed as a carrier of values, cultures, and ways of 
life, it may be viewed as another form of colonization and can result in the 
loss of national traditions and uniqueness. 

The effects of globalization (such as increased mobility, the exploitation of 
information and communications technologies for distance learning, etc.) 
call into question the continued relevance of national boundaries for many 
of the processes studied by the international statistician. 

“Probably the best answer to the challenge globalization puts to 
statistical offices is to do what companies do: they form all kinds of 
world-wide alliances and re-distribute tasks among the constituent 
partners in order to save costs and to increase productivity, we should 
follow suit and think much more seriously about an international 
network of NSIs [national statistical institutes], national central 
banks, and international statistical offices, which collect the data 
where it is most appropriate, share the information with each other, 
and perhaps specialize in accordance with local circumstances and 
expertise” (Keuning, 2000). 

Is this way of proceeding correct? Is the pooling of nationally gathered 
data the right strategy, or is a radical reappraisal of the international 
statistical system required for obtaining a more comprehensive global 
picture? Do we over-emphasize the national? 

2.4. The Injormaiion and Commanicaiions Technologies 

Alongside globalization, the increasing penetration of information and 
communications technologies (the ICT’s) is also presented as a means of 
reducing inequities. Indeed, as the example of India shows, ICT can 
generate employment and wealth within a developing country, provided that 
there is foreign investment, economic liberalization, and a pool of technically 
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educated people. However, ICT can only deliver benefits if an adequate 
telecommunications infrastructure is available and if people have access to 
equipment and training. Fewer than 6 percent of people worldwide own a 
computer, and there are more computers in the United States than in the 
rest of the world put together. Some 94 percent of all Internet users live in 
the 40 richest countries, and 30 percent of Internet users have a university 
degree. It would cost the average Bangladeshi more than eight years of 
income to purchase a computer compared with one month of salary for the 
average Americam. There are more telephones in Tokyo than in the whole of 
Africa, and more than half of the people in the world have yet to make a first 
telephone call. 

For developing countries, which are always characterized by a shortage 
of trained teachers (a problem exacerbated in some countries by the ravages 
of AIDS on the teaching profession - in Zambia the mortality rate amongst 
teachers is over 70 percent higher than among the general population, 
1,331 teachers having died of AIDS in 1998 thus putting a great strain on 
the education system), new modalities must be sought to overcome 
deficiencies in the provisioning and quality of education. In this context, the 
ICTs are important not only as a means of linking the underprivileged to the 
global economy but also as a meams of delivering education to them. As the 
Assis tan t Director-General of UNESCO for Education, Sir John Daniel, has 
written: 

I see the trend towards open-source courseware as extremely helpful 
to institutions in the developing world. Good distance learning 
material, whether delivered on the Web or through other media, is 
expensive to develop. It would be tragic if, through excessive royalties 
for its use, courseware became another area where there is a net flow 
of wealth from the poor world to the rich world. Open-source 
courseware will help to bridge the digital divide and bring the day 
closer when all the world’s population can exercise their human right 
to an education (Daniel, 2001). 

Initiatives taken by the Massachusetts Institute of Technology (MIT) in 
the USA to put its course material in the public arena may result in a wave 
of developments in this area. As ICT forges new ways of delivering higher 
education, it becomes necessary to rethink the statistical measures used to 
capture data on access, participation, and achievement. 

3. DEVELOPMENT GOALS 

In 1995, development ministers from the member countries of the OECD 
Development Assistance Committee (DAC) determined to review past 
experiences £md the planning of policies into the next century. The resulting 
report. Shaping the 21®‘ Centanj: the Contribution of Development Co- 
operation (OECD, 1996) included seven specific goals, which have become 
known as the international development goals, and argued that their 
achievement would be of profound significance for future generations. The 
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goals were drawn from a series of United Nations conferences on key 
themes: education (Jomtien, 1990), children (New York, 1990), the 
environment and development (Rio de Janeiro, 1992), population and 
development (Cairo, 1994), social development (Copenhagen, 1995), and 
women (Beijing, 1995). The seven, however, are only a subset of the goals 
about which agreement was reached at these other conferences. 

The international development goals have been endorsed by seventy- 
seven African, Caribbean, and Pacific countries as part of the Cotonou 
agreement^ and by OECD/DAC, the World Bank, and the IMF. 

In September 2000, 149 heads of state at the United Nations Millennium 
Summit endorsed the Millennium Declaration (UN, 2000a) which includes a 
set of goals which closely resemble the international development goals. 
However, there are some key differences relating to the exclusion of a 
reproductive health goal (as a result of veiy difficult negotiations lasting 
nearly a month), the inclusion of a goal concerning disease (aimed at halting 
and then reversing the spread of HIV/AIDS, malaria, and other diseases by 
the year 2015), and another relating to “cities without slums” (aimed at 
achieving “a significant improvement in the lives of 100 million slum 
dwellers by 2020”). These are known as the Millennium Declaration Goals. 

The Millennium Declaration also has a number of other resolutions that 
are framed as goals. However, not all of them are easily turned into 
measurable targets, e.g., “to provide special assistance to children orphaned 
by HIV/AIDS”, “to develop strong partnerships with the private sector”, “to 
implement strategies that give young people everywhere a real chance to 
find decent and productive work”, and “to encourage the pharmaceutical 
industry to make drugs more widely available”. 

Discussing development goals, even when they do not yet address issues 
of higher education, is needed because the goals are increasingly being 
turned into indicators which are then used for drivers of statistical systems. 
In order to increase the emphasis on higher education, higher education 
must get itself included in these agendas. 

Concerns are often expressed about the burden of data collection upon 
countries, and the implication drawn tliat the use of priority indicators 
reduces this burden. Certainly international agencies should be moderate in 
their requests for data. They should consult extensively with countries and 
should provide justification for their requests. Stressing the disadvantages 
of data collection (the burden), rather than the value of the data, creates a 
vicious spiral. Experience shows that financial, political, and institutional 
support for statistics is generated when they are used extensively and are 
seen to make a difference in regard to the quality of policy decisions. Under 
these circumstances, users both inside and outside Government, will be 
important and persuasive allies in the fight for their continued collection. 



2 See <hap://europa.eu.inl/conim/develo|)nienL/coLonou/agreement_en.hlni>. 
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4. DEVELOPMENT INDICATORS 
4. i . Translating Goals into Indicators 

Considerable work has taken place in a variety of different fora to translate 
development goals into indicators. At a specific level, much of this work has 
been carried out within specialized agencies of the United Nations. In 
addition to sets of indicators developed in response to specific World 
Conferences, other sets have been created for particular publications such 
as the various World Reports including the influential Human Development 
Report of the United Nations Development Programme. 

Some efforts at developing a cohesive set of indicators should be noted. 

4.2. A Common Framework for Indicators 

At a more general level, the United Nations is developing a common 
framework for development assistance, known as the United Nations 
Development Assistance Framework fUNDAF). UNDAF incorporates a review 
of the measures to translate the commitments made by countries at global 
conferences into country level action plans following consultations on 
national priorities, the identification of key indicators and data collection, 
and implementation, monitoring, and evaluation. Thus, a set of “common 
country assessment indicators” has been developed. 

In addition, the United Nations Statistical Commission has developed a 
Minimum National Social Data Set to provide a focus for achieving a 
manageable set of core indicators of progress in social development. 

The foregoing attempts to make some sense of this maze of indicator 
initiatives, but it is not surprising that the Economic and Social Council of 
the United Nations requested that a committee be established to address 
concerns about the proliferation of statistical indicators and the demands 
that this proliferation was placing upon national statistical systems. The 
report of the council argues for a coherent set of priority indicators. In 
addition, in March 2001, the United Nations Statistical Commission set up 
a “friends of the chair” group, under the leadership of Professor Tim Holt, to 
examine different indicator sets and to make recommendations on 
consolidation and co-ordination. However, there are some who believe that 
the problem of inconsistencies among sets of indicators would be solved if 
: only everybody were to rally behind the Millennium goals and to use these 

to determine a definitive set of indicators, which do not include goals on 
higher education. 

4.3. How Important Are Indicators? 

The high level political significance attached to various international goals is 
to be welcomed by statisticians and can hopefully be turned to the 
advantage of weak statistical systems throughout the world. And we need to 
acknowledge that there are many weak statistical systems. Since the 
batteries of indicators are dependent upon achieving responses from as 
many countries as possible, with keen attention being paid to the poorest 
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countries, the gaps in data can be highlighted and resources sought to 
assist those countries in collecting the relevant data. The data are certainly 
very powerful for advocacy purposes and enable the international 
community to co-ordinate actions. In addition the targets can be very useful 
for purposes of accountability. 

Whilst recognizing the value of indicators and warmly welcoming 
initiatives to reduce the multiple sets of indicators into a priority selection, 
they should not be regarded as encapsulating the whole contribution of 
statistics. The dangers are twofold; 

First, indicators should not have more meaning imparted to them than is 
justified by their content. Mark Malloch-Brown might have fallen into this 
trap, when he noted that. 

In Brazil, President Cardoso has targeted a $7.4 bm anti-poverty 
programme at the 14 states with the lowest Human Development 
Index. In Madyha Pradesh, India, the chief minister, Digvijay Singh, 
recently described to me how he governs by his human development 
index (Malloch-Brown, 2001). 

Second, to reduce the role of statistics to mere monitoring and advocacy 
is to ignore their value in formulating evidence-based policies. Work on 
indicators should therefore be complemented by support for statisticians to 
develop rich databases about the state of societies at a fundamental level. 
There is an impression that the priority indicators are the only statistical 
outputs a country needs even for its own governance. A report by the World 
Bank task force on higher education in developing countries recommended 
in 2000 that. 

Each developing country make it a national priority to debate and 
deterrnine what it can realistically expect its higher education system 
to deliver. The debate must be informed by historical and comparative 
knowledge about the contribution to social, economic, and political 
development - but should also take account of the challenges the 
future will bring. It should establish for each higher education system 
clear goals that policymakers can use to view the higher education 
system as a whole, determining what each part can contribute to the 
public good. 

This kind of holistic analysis of higher education systems has 
rarely been attempted. It does not mean reverting 4o centrally planned 
systems - far from it. Instead it offers the ability to balance strategic 
direction with the diversity now found in higher education systems 
across the world (The World Bank, 2000). 

4.4. Attributes of Indicators 

Goals chosen without any reference to their measurability or other 
attributes are often, through the mechanism of World Conferences, 
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endorsed by countries before statisticians have had an opportunity to 
question their characteristics as follows; 

- Are they realistic? 

- Can they be measured? 

- Will they show sufficient change over time to be valuable as 
yardsticks? 

- Are relevant benchmarks in place? 

- Are they universal, and do they make sense in different parts of the 
world? 

- Are they coherent with what has been used before? 

After such “declarations”, goals cannot easily be changed. Sometimes the 
translation of goals into indicators is not viewed as a statistical process at 
all - statisticians being perceived simply as technicians who take indicators 
as given and organize the data collection but do not have any relevant 
expertise with respect to the creation, determination, or definition of 
indicators. Too rarely is there recognition that conceptual development and 
fieldwork are needed to develop reliable indicators. Since indicators are 
often expressed in terms of change over time [e.g., halving adult illiteracy), 
the unquestioned assumption that there are adequate baseline data can be 
very problematic. 

The selection of appropriate goals is critical since, ideally, they should be 
realistic and achievable. It has been proposed that countries should be 
stretched to achieve them, but if goals are too difficult to achieve and are 
thus not achieved, the result is detrimental, as it perpetuates a sense of 
failure. It is not easy to see how such goal achievement can be managed 
with a set of identical indicators for all countries of the world, which are of 
course at very different stages of development. Skeptics of the development 
indicators programme often cite the fact that indicators have been produced 
as a result of World Conferences over many years and that all the targets 
have consistently been missed. 

Because of the need to collect the same data for all (or a large group of) 
countries, the indicators must reflect the lowest common denominator. 
However, inertia in the system, together with anxiety that no changes 
should be made to the agreed set of indicators, has an unfortunate 
consequence in that countries are not allowed to grow in their statistical 
expertise. Such inertia can dissuade countries from developing new 
indicators and entrench the selection made at a particular point in time. 
Incidentally, changing an indicator is often confused with “changing the 
goal posts” even though the indicator may have been a very inadequate 
proxy for the goal. This perception is particularly acute in relation to higher 
education for which acceptable universal indicators have yet to be 
developed, and the world is changing very rapidly in regard to the 
diversification of education systems. 
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4.5. Presentation and Use of Indicators 

How indicators are presented can be a cause for concern. They are often 
stripped of their essential meta-data, which countries and agencies have 
been at great pains to collect and which are essential for informed use. One 
cannot make sound use of data without having a good understanding of the 
political, economic, and social context of the given country. Many indicators 
are defined as ratios with the numerator and denominator coming from 
different sources and even from different agencies within countries. This 
way of presenting can result in incompatibilities in the data and significant 
errors in the resulting indicators. Moreover a change to either the 
numerator or denominator (which can happen when a Census is carried 
out, for example) can mean that the indicator changes markedly owing only 
to measurement error. 

A significant rise in the use of performance indicators in higher 
education has been noted. Such indicators have a number of purposes: 

- to provide more reliable information on the nature and performance of 
higher education; 

- to allow comparisons to be made between individual institutions: 

- to enable institutions to benchmark their own performances: 

- to inform policy: 

- to contribute to the public accountability of higher education. 

Unfortunately, indicators can easily be fed into league tables. Such 
tables are prone to misinterpretation resulting from the lack of appropriate 
contextualization of outcome indicators, inadequate specification of the 
statistical models, and conceptual problems in devising measures of “value 
added”, the fact that the “value added” is assumed to be meaningful despite 
the fact that input and output data are often measured in different units, 
and the failure to take account of uncertainties in the data. 

Increasingly, indicators at countiy level are being used in league tables, 
too. The media triumph of the UNDP Human Development Index has 
encouraged other agencies to create league tables with their own selections 
of indicators. However, in such international league tables, data from veiy 
different countries are incorrectly assumed to be of equivalent quality, and 
key indicators may be manipulated or withheld by countries to alter their 
V rankings with detrimental effects on the quality of the data (a problem 
which is especially acute when a high level of attention is given to particular 
league tables). Furthermore, such league tables sometimes use composite 
indicators that have been created by combining data across totally different 
topics. Whether or not the creation of these league tables is a statistical 
exercise is debatable: however, the fact is that it is widely perceived as such, 
and this perception can affect the ability of statisticians to establish 
credibility for data. International statisticians have sometimes been 
reluctant to elaborate on the limitations of league tables because they can 
see that the media and the political attention given to league tables raise the 
profile of development. 
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Even though nations are of widely varying sizes, and many of them have 
very decentralized decision-making, the emphasis is mostly on indicators at 
the national level, impljdng that the nation is the only relevant unit of 
analysis. This approach persists despite the fact that many decisions are 
made at sub-national level and that wide variations exist across many 
countries. The focus on nations concentrates on totals or averages, whereas 
it is often the distribution within countries that is of interest. Inequality in 
the world is badly under-estimated through an over-emphasis on national 
data. 

However imperfect they may be at reflecting goals for development, 
indicators have gained credence, and the fact that they are inevitably an 
over-simplification gets conveniently forgotten. Any issue, which is not 
measured by an indicator, is seen as unimportant, and conversely, any 
issue with an associated indicator is automatically assumed to be 
important. 

Economists hove come to feel 

What can’t he measured isn’t real. 

The truth is always on amount 

Count numbers, only numbers count. 

Indeed among the most debilitating characteristics of life in deprived 
situations is a loss of dignity which affects the capacity to escape from the 
cycle of deprivation, but is not susceptible to quantification. 

5. INTERNATIONAL COMPARABILITY 
5. 1 . Purposes of Comparabtlity 

A significant issue that the “Roundtable on System-Level Indicators for 
Tertiaiy /Higher Education” addressed was the fostering of the collection of 
comparable data across nations, the main objectives being: 

- To enable countries to gain a greater understanding of their own 
situation by comparing themselves with others, thus learning from 
one another and sharing good practice. “They (cross-national 
measures) help to reveal not only intriguing differences between 
countries and cultures, but also aspects of one’s own country and 
culture that would be difficult or impossible to detect from domestic 
data alone”. 

- To permit the aggregation of data across countries to provide a global 
picture, thus enabling the design of international initiatives informed 
by evidence. 

- To provide information for purposes of the accountability of nations 
and for the assessment, development, and monitoring of supra- 
national policies. 

Although collecting comparable data is difficult, the measurement 
problems encountered are not a sufficient reason for abandoning cross- 
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national research. The importance of the latter is growing owing to 
globalization and moves towards cross-national governance. Instead, we 
must argue for circumspection in the analysis of comparable data and a 
resistance to drawing “heroic conclusions on the basis of unexpected 
national variations in the answers to a single question.” Jowell (1998) 
outlines a number of practical rules for mitigating some of the problems in 
relation to comparative research: “Social scientists should undertake not to 
interpret data relating to a country about which they know little or nothing”. 
“Analysts of cross-national data should resist the temptation to compare too 
many countries at once”. And “Cross-national surveys should ideally be 
confined to the smallest number of countries consistent with their aims”. 
On the other hand, the call to analyze data whilst taking account of the 
appropriate context, to provide access to full methodological information for 
each nation, and to suspend belief in any major country differences 
discovered before determining whether they result from differences in 
methodology or the interpretation of methodology are all very important 
lessons for international statisticians. As Jowell points out, all “quantitative 
surveys depend for their reliability on a sort of principle of equality or 
equivalence”, and thus the problems are not unique to cross-national 
studies. 

5.2. Models for Achieving Comparability 

There are substantially different models of collection of cross-national data. 
These include: 

- the (fortunately discredited) “safari method”, whereby international 
researchers visit countries to collect the data they need, withdraw the 
data to an industrialized country, and analyze it there with limited 
appreciation of the context of the data; 

- the collaborative model, as exemplified by the International Social 
Survey Programme, in which a research team comprising participants 
from ^1 the countries jointly design the key aspects of the study; 

- pre-collection harmonization, whereby representatives of the countries 
are consulted to resolve differences in their methodologies in advance 
of the data collection, and the work is mediated by an international or 
regional statistical agency (this model is employed by EUROSTAT in 
relation to much of the cross-European data); 

- post-hoc harmonization, in which countries collect data independently, 
with the data being later re-analyzed in the light of comparative 
research (United Nations, 2000b). 

Obviously, it is desirable to involve national experts who have an 
understanding of the individual data sets and their contexts and can 
interpret them. However, this way of proceeding is expensive and may not 
always represent the best use of scarce resources. A compromise model 
might call for work in regional teams, which collaborate closely, and then 
attempt post-hoc harmonization of the data across the regions. Such work 
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might be more efficient because countries can more easily and cheaply 
share expertise. They will have a greater understanding of one another’s - 
situation. They are more likely to share common problems and to find 
similar relevant solutions. The diversity of languages may be reduced, and 
the “leading country” in a region may help the others. The countries may 
share a common identity. In the higher education field, there is an immense 
cimount to learn from the OECD experience in developing indicators. 

But the regional model also has disadvantages in that it is more difficult 
for the leading countries in a region to develop and to learn. The regional 
model can lead to a loss of the global picture. Partitioning the world can 
lead to a fragmentation of policies and a lack of appreciation of the 
interdependence of countries. For example, it can be argued that equity and 
sustainability within the poorest countries cannot be achieved without a 
fundamental rethinking of northern consumption and expenditure. 
Allocating countries to homogeneous clusters also reduces the opportunity 
to explore exciting differences that emerge when making comparisons 
across a heterogeneous group. For an example of heterogeneous 
comparisons in the field of education, see Colclough (2001), who has carried 
out a study comparing the very different education systems in Anglophone 
and Francophone Africa. 

5.3. Problems in Achieving Compcwability 

The problems involved in collecting comparable data are similar to those 
outlined on indicators; that the need for cross-national data leads to the 
acceptance of the lowest common denominator: that the latter fosters 
inertia in the system by making revisions difficult (the problem of making 
the methodologies relevant for the strongest countries as well as for those at 
the “trailing edge”); that it is vital to be sensitive to the burden of data 
requests on countries: and that cross-national data may not be specific to 
national needs thereby making it more difficult to involve users of national 
data and distorting national agendas. For example, data harmonized 
internationally are of less value nationally because age ranges for different 
stages of education vaiy among countries. Thus internationally harmonized 
education statistics will not necessarily correspond to the picture recognized 
in a particular country. 

Cross-national comparative analysis is an important statistical tool, but 
it also brings risks: frequently politicians or senior civil servants are 
dismayed by the relative results for their countries and tend to blame the 
messenger rather than examining the message. An escape for statisticians 
is to direct the blame onto the methodology. No cross -national study can be 
perfect as far as comparability is concerned, and it is only too easy to find 
reasons why the data should not be taken seriously. An outcome of this 
unhappy state of affairs is the withdrawal from cross-national research of 
those countries that achieve disappointing results and occasionally a more 
catastrophic impact upon the careers of the national statisticians involved. 
On the other hand, some statisticians manage to turn poor comparative 
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results to their advantage using them as a “rallying call for more resources”. 
International statisticians, whilst being sensitive to the situation and 
concerns of their peers in national agencies, should fight for the 
transparency of comparative data and should not collude in their 
suppression when the data are uncomfortable. Transparency should, of 
course, extend beyond the data themselves to cover information about the 
methodology. 

Harmonization of data collection is necessary but not sufficient to 
generate valid cross-national data. Full access to the individual components 
is also essential. For example, within Europe access to cross-national 
sources can be hampered by lack of unified legislation on confidentiality 
and data protection. 

6. INTERNATIONAL CLASSIFICATION OF DATA 

The collection of comparable data must be conceptually well anchored and 
is heavily dependent upon the use of standardized classifications of key 
variables. A major part of the work of an international statistician is the 
development and maintenance of such classifications. Without such 
activities, “comparability is only skin deep” (Church, 1996). Moreover, the 
consultation necessary to develop classifications is expensive and time 
consuming and, even when standard classifications are employed, 
differences among countries can be an artifact of the data collection method 
or can reflect the administrative system of which they are a product and 
thus not be “real”. 

Classification activities have been of prime importance... not because 
of some imagined obsession amongst statisticians with order but 
because of an indispensable need for coherence in the statistical 
description of economic and social realities.... It is thanks to the 
universality of this language that information from Sweden or 
Portugal, Australia or Mexico can express the same economic and 
social phenomena in a comparable way.... To be effective, a statistical 
language must be developed systematically so that different kinds of 
norms are compatible and relations can be established between 
different information (Malaguerra, 2000). 

Holt (1998) argues that 

It is rare for the concepts that we strive to measure to be driven by a 
well-defined theoretical construct. When conceptual clarity is lacking, 
statistical integrity can nevertheless be enhanced by adherence to 
international standards.... The very fact that such standards are 
agreed internationally is both a guide and a support for national 
statisticians producing statistics whose interpretation may be 
politically sensitive. 

Whilst both Malaguerra and Holt are correct with respect to the 
importance of developing strong international standards and classifications. 
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there is a mismatch between the expectations and the resources at the 
disposal of those working on standards which often means that the . 
standards are the objects of insufficient research and are based on 
inadequate consultation. 

7. IMPROVING THE QUALITY OF INTERNATIONAL AND NATIONAL 
STATISTICS 

7. 1 . Defining Quality 

Quality is defined in ISO 8402 as “the totality of features and characteristics 
of a product or service that bear on its ability to satisfy stated or implied 
needs”. 

One role of statisticians is to improve the quality of collected data. 
However the term, quality, as applied to official statistics, is not easily 
defined, being comprised of many components including: 

- validity: 

- reliability: 

- relevance to policy: 

- potential for disaggregation: 

- currency: 

- punctuality: 

- coherence across different sources: 

- clarity and transparency with respect to known limitations: 

- accessibility and affordability: 

- comparability through adherence to internationally agreed standards: 

- consistency over time and space: 

- efficiency in the use of resources. 

The optimum combination of these components is dependent upon the 
use to be made of the data. Data acceptable for one purpose might be 
inadequate for another and, since most data are used for many different 
purposes, the process of determining “fitness for purpose” is extremely 
complex and requires wide consultation. Inevitably, the trade-offs to balance 
the different components of quality will lead to the disappointment of 
certain users. 

7.2. Achieving Quality 

International statisticians are constrained in what they can do about the 
quality of data received, and since the expectations of users are often 
unrealistically high, these constraints must be honestly expressed. So what 
options are available? 

The fundamental and overriding aim must be to collect data which are 
relevant to the policies being addressed, to ensure that there is agreement 
on the scope and definitions being used, that providers of the data are 
partners in the process, and that they understand the importance of 
collecting high quality data. Increasingly, the problem in relation to data on 
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higher education is that there is a need to reflect private as well as public 
sources of education provision and funding. 

- Consult countries about data collection. It is important that the data 
requested from countries be consistent with, and anticipate changes 
in, their national priorities as well as taking account of their 
capabilities to meet the requests. (However worthy this principle, it is 
in reality, expensive and time-consuming, especially for truly 
international agencies to develop effective means of consultation, and 
inevitably a degree of tokenism is rife. Statisticians in regional 
agencies dealing with more homogeneous groupings of countries are 
better able to take account of the views of their national counterparts.) 

- Engage in partnerships with both users and producers of data within 
countries. The objectives are to develop a greater understanding of the 
significance of continuous improvement in statistical work, to 
examine ways in which quality might be assured, especially through 
the organization of national statistical systems, and to forge national 
commitments to provide quality data. In addition, national 
statisticians should be assisted in improving their user focus and 
communication skills. 

- Be temperate in what data are requested. There is a danger that the 
sheer number of data collection exercises undertaken by international 
agencies could overwhelm the capabilities of some countries to 
respond. Therefore, each new initiative must be subject to careful 
scrutiny, and there must be periodic reviews of existing activities. 
(However, the latter can be difficult for the international statistician 
who wishes to be responsive to the wide range of reasonable demands 
for cross-national data.) 

- Identfy “key statistics”. A possible means of addressing the problems 
resulting from the wide variation among the statistical capacities of 
countries is to prioritize international demands so that those with 
limited resources may concentrate on key data collections. 

- Minimize revisions to intemationcd surveys, classifications, and 
methodologies. (This stipulation inevitably conflicts with the need to 
obtain data relevant to current needs, and achieving a balance 
between revisions and continuity is made difficult by the different 
stages of statistical development of countries. For example, the 
UNESCO Institute for Statistics is under pressure from statisticians in 
some developed countries to revise the 1997 International Standard 
Classification of Education (UNESCO, 1997) which many countries 
are still in the process of implementing.) 

- Ensure that good practices are employed within the international 
agencies. These practices should include openness about the 
collection, processing, and analytic methods employed, especially in 
relation to estimation and imputation procedures as well as the 
importance of mentioning in accompanying text any uncertainties in 
forecasts or projections, the need for time series to be provided on a 
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comparable basis, the importance of building in-house expertise and 
using advisory /review bodies, the value of closer co-operation with the 
suppliers of key data, the strengthening of quality control procedures, 
and the inclusion of source information for all statements based on 
statistical evidence. 

- Develop a framework for the delivery of irformaiion on the quality of 
data. There is a legitimate concern about the spurious accuracy of 
data, which has come to be known as the “professional culture of 
precision”. As Thomas Mayer relates, we “act like the person who, 
when asked about the age of the Amazon river, replied that by now it 
is one million and three years old, because three years ago he was 
told that it was a million years old” (Mayer, 1992). 

- Adopt the principle of “no surprises”. National statisticians should be 
informed if data have been altered in any way in advance of 
international publication, thereby giving them (where feasible) the 
opportunity to comment on the changes. (There are those who argue 
that under no circumstances should international statisticians 
publish any data that have not been approved by the appropriate 
authorities within countries. This view is held for two reasons: one 
that “filling in data gaps” discourages countries from investing in 
statistical systems which provide the complete set of data; the other 
that this right of veto is a key element of “data ownership”. However, 
there is an inherent conflict between this right and the need to 
provide better estimates in cases in which the national data are 
known to be wrong, especially if there is suspicion that the data have 
been politically manipulated.) 

- Share data and irformation, including meta-data and methodologies, 
across international agencies. In current jargon, this recommendation 
might be described as “joined-up international government”. It 
reduces the burden on countries of responding to requests for data 
and subsequent clarifications and enables the creation of harmonized 
cross-national data sets minimizing problems caused by contradictory 
data sources. (Even though good progress has been made under the 
auspices of the United Nations Advisory and Co-ordination Committee 
on Statistical Activities to reduce the duplication of requests for data, 
it has not been entirely eliminated. One of the obstacles to co- 
ordination is the decentralization of statistical activities within many 
international agencies so that surveys and other statistical activities 
are often conducted by staff members who do not work in the 
respective statistical unit. Sharing meta-data across agencies and 
collaborating in data processing are not yet universally accepted 
practices, and the mantra, “collaboration on input, competition on 
output” has not proved helpful). 

- Strive nationally and internationally to improve public confidence in the 
integrity and independence of data. “Public trust is the crucial test for 
the quality and integrity in official statistics”. The influential report of 
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the Royal Statistical Society in 1990 made a powerful case for 
strengthening public confidence: “If statistical information is not seen 
to come from an authoritative, independent source, it loses its value 
as a currency of public debate, and confidence in the debate, itself will 
be eroded”. 

- Embed data requests within programmes of relevant statistical capacity 
building. The abilities of countries to respond to international data 
requests should be assessed. Assistance should be provided through 
programmes of capacity building. These should include the 
identification of national centers of excellence where statisticians may 
share their experiences and good practices. 

8. DEVELOPING THE STATISTICAL SKILLS BASE IN COUNTRIES 
8.1. The Importance of Capacity Building 

The aim of statistical capacity building should be to help countries to 
become self-reliant, both financially and institutionally, so that they may 
acquire the expertise to determine their own data needs and priorities, to 
collect these data, to interpret and to use them effectively; to undertake 
research, problem solving, and problem formulation; and to sustain these 
capacities. 

Capacity building should not be confined to the staff of the national 
statistical institutes but should also take account of the needs and 
circumstances of statisticians in line ministries or in local government who 
often play a critical part in data collection yet can be more vulnerable to 
political interference in their work and often do not see themselves as part 
of a professional community. Although there is an apparent 
acknowledgement of this situation, in practice, almost all-statistical 
capacity building focuses on the staff of National Statistical Institutes. 

The development of a professional identity for statisticians can be a very 
valuable aspect of technical assistance. Consideration should also be given 
to ways in which researchers and policy analysts might be helped to gain 
access to, to utilize, and to value the data of their own countries. Much 
more attention should be given to renewing and sustaining strategic centers 
of potential excellence in the developing world, which have been depleted by 
over twenty years of South-to-North brain drain. 

Partnerships in Statistics for Development for the Twenty-First Century 
(PARIS21) - an initiative of the World Bank, the IMF, OECD, and the United 
Nations - was launched in Paris at the end of 1999. It aims at building 
statistical capacity as the foundation for effective development policies by 
helping to develop well-managed statistical systems that are accorded 
appropriate resources. In the longer term, it hopes to promote a culture of 
evidence-based policy making and monitoring in all, but especially poor, 
countries in order to serve to improve transparency, accountability, and the 
quality of governance. These laudable aims now deserve to be made 
concrete by means of a positive action plan backed by appropriate funding. 
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8.2. Models of Capacity Building 

In September 1997, a workshop held in the Netherlands brought together' 
donors and recipients of technical co-operation in statistics to discuss 
w ic models of assistance are most helpful. The outcome was a very useful 
proposal for “guiding principles for good practices in technical co-operation 
in statistics’ which was subsequently discussed and adopted by the UN 
Statistical Commission (United Nations, 1998). These guidelines address the 
tact that there are responsibilities on the part of the recipients - particularly 
in relation to their commitment to the assistance, the availability of 
motivated staff, and their absorption capacity - as weU as the donors, to 
ensure that full consultation takes place and that both have a clear grasp of 
the needs and priorities of the given country in order to design appropriate 
assist^ce. Karlsson (1997) argues that capacity, like democracy, cannot be 

created from the outside and that “essentially it is formed by internal 
d3mamics”. 

8.3. Difficulties in Capacity Budding 

A recurring theme of the country participants at the workshop in the 
Netherlands was the high cost of international consultants. The anguish 
caused by the fact that a consultant might be paid twenty times more than 
the staff being assisted can be very destabilizing. This concern is evident in 
the development literature. Jaycox (1993) writes of the deleterious impact 
on professionalism in developing countries of the tendency to use expatriate 
technical assistance to solve all kinds of problems. He called for the creation 
of a “demand for professionalism in Africa”. 

Ko Chih Tung has argued that past development efforts regarding 
education statistics have “led to heavy reliance, sometimes near total 
dependency, on foreign experts and imported turnkey solutions, which have 
often resulted in ignoring the importance of directly involving national 
producers and consumers of statistical services. In this mode, development 
TO-operation has often been narrowly technical and therefore self-limiting.” 
He describes problems of redundancy, incompatibility, and even a mutually 
corrupting dependency between the providers and receivers of statistical 
assistance. He speaks powerfully of the “graveyard of collapsed, abandoned 
and outdated systems” (Tung et al., 1999). 

9. SHARING DATA 

9.1. The Importance of Sharing Data 

There is widespread recognition of the importance of exploiting, in social 
research and analysis, the rich data resources of official agencies in 
particular but also of academic and commercial organizations. In relation to 
data on higher education, this source is especially important because those 
mvolved in higher education must be partners in its analysis. However, the 
tacilities for data access in the developing world are veiy poor. We need to 
acknowledge the importance of giving access to data in electronic form 
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(alongside conventional published material) and, where legal constraints 
permit it and operational procedures can be devised, to individual level data 
as well as to aggregate data. We must also help statisticians in the 
developing world, both users and producers, to reap the benefits of sharing 
data. Tliese benefits can be seen in terms not only of efficiency through the 
direct exploitation of the data but also of altruism, with existing data sets 
being used for educational and training purposes as well as knowledge 
transfer. The creation of a community of skilled users, who will help to 
improve the quality, credibility, and visibility of the data, can directly benefit 
the data provider. It will reduce the response burden, which is of particular 
concern to developing countries. 

9.2. Fostering the Sharing of Data 

We need to create a culture in our institutions and in our societies, more 
generally, in which data sharing is the norm. A prerequisite for this is a 
climate of openness in which criticism of data collection or analysis is 
factual and temperate, with data being used responsibly. Even within the 
academic social science sector in the developed world, there are pockets in 
which the culture of data sharing is not accepted, perhaps because primaiy 
researchers are concerned that they might not receive formal credit for data 
sharing and that other academics might generate earlier or better 
publications from their secondary research. The institutional reward system 
for research needs to be examined to identify and remove such barriers to 
sharing. It is unfortunate that developing countries are being encouraged to 
move from a collegial higher education system to one that is more 
competitive without a full understanding of some of the negative benefits, 
which could be especially acute in countries with limited resources. 

We should seek to develop data policies to ensure that deliberate 
replication is encouraged but that ignorant duplication does not happen, 
and to exploit investments in data. 

The case that data resources for social researchers are analogous to large 
expensive pieces of equipment for physical scientists was accepted by the 
European Union for the Fifth Framework Programme of funding. Despite 
this growing recognition of the value of data, there are many areas of the 
world in which data are simply not available for purposes of research, 
’ because of a poor infrastructure and limited expertise in data handling but 
also because of the weak links between government officials and academics. 
This situation must be of concern since the effect is a widening gap between 
the developed and the poorer countries in analytic skills, which exacerbates 
the disparities. 

The development of endogenous capabilities is an effective deterrent to 
brain drain. The UNESCO report on the World Summit for Social 
Development (1996) states that 

Alongside the action to enhance national and regional capabilities for 

higher education and scientific and technological training, it is also 
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essential to promote both basic and applied scientific research and 
the dissemination of its results. 

Developing analytic skills within countries, which are complementary to 
those within the international agencies, is essential to partnerships, and 
access to information is a necessary prerequisite. The capacity of countries 
to attract and retain high-level professionals is critical. As an international 
community, we must examine and tiy to reduce the barriers that prevent 
countries from acquiring and maintaining a cadre of statistical experts. 

10. FACILITATING ACCESS TO DATA 

Support must be given to enhance access to information and 
communication technologies and to ensure that wider use is made, by 
researchers in the countries concerned, of the data that have been 
generated with the very precious resources of given countries. Tlie lack of 
preseiwation facilities and expertise means that a small trickle of valuable 
data is often lost, and the lack of accepted systems for the involvement of 
academics means that even the few existing data are not exploited. The 
lessons that electronic data are not a finite exhaustible resource and that 
their value is increased, not diminished, by their use have not yet been 
learnt in much of the developing world. 

11. DILEMMAS AND PRINCIPLES 

11.1. National Sovereignly versus National Accountability 

A theme mnning through this study has been the tension between respect 
lor national sovereignty and the importance of ensuring access to national 
data of integrity. In particular, data are a vital part of the system of 
accountability in relation to the use of overseas assistance. Considerations 
of a political nature (establishing or maintaining alliances, sustaining 
political blocs or spheres of influence, ensuring votes in international 
organizations) are yielding to issues such as good governance, reducing 
inequalities in society, social development, and promoting the rights of the 
citizen. 

11.2. Fundamental Principles oj Official Statistics 

Official statistics are essential for obtaining a transparent picture of 
societies and how they are developing across the world. They also serve as a 
basis for the efficient and appropriate implementation of political decisions 
and for the effects of these decisions to be monitored. In recognition of the 
importance of establishing national statistical institutes that are capable of 
providing data of integrity, the Conference of European Statisticians 
adopted the Fundamental Principles of Official Statistics in 1992. These 
Principles were devised in part to assist in the creation of national statistical 
institutes in the new environment of post-communist Eastern and Central 



INDICATORS OF HIGHER EDUCATION • 59 



Europe, underpinning the moves to more democratic systems (United 
Nations, 1992). 

The Fundamental Principles have served to “foster a common 
understanding about the importance of statistical integrity. They have 
raised consciousness, and in so doing have strengthened international 
bonds. In addition to the countries in transition, others have also been 
prompted to look afresh at their own frameworks of statistics to see how 
they stand up to scrutiny against the principles”. 

These Principles originated in Europe but have also proved useful in 
other parts of the world, as demonstrated by their adoption by the UN 
Statistical Commission in 1994. As Seltzer (1994) explains, “The 
Fundamental Principles, United Nations handbooks, and similar materials 
were designed to aid statisticians, politicians, and the public in each 
countiy to build a useful, impartial, and reliable statistical system. 
Although individual national statisticians might themselves invoke the 
Fundamental Principles or a United Nations handbook to protect the 
integrity of the statistical system when it is subject to domestic threats, the 
same statisticians may well react defensively if they are seen as the subject 
of some sort of international investigation”. 

1 1 .3. Ethics and Codes of Practice 

The Need for Ethical Frameworks 

All of us, regardless of our chosen professions, have a duty as humans to 
exploit our skills, expertise, and opportunities in pursuit of justice and 
human rights. Openness about methods and operations is a prerequisite for 
building confidence in statistics. A statistical agency which shows a 
balanced and open approach to the measurement of its own performance 
can only serve to strengthen its reputation for objectivity and impartiality - 
even when some performance measures are not as positive as might be 



A broader but related issue is whether or not statisticians accept 
responsibility for acknowledging the limitations of quantitative information. 
As statisticians are wont to do, they over-emphasize the quantitative. Richer 
sources of material might be created were statisticians to appreciate more 
: readily that quantitative material alone provides a partial picture and to 
work in partnership with social scientists to add qualitative, in-depth 
research to the quantitative frame. They must be more willing to enter into 
discussions about the strengths and limitations of their studies. One of the 
criticisms is that they oversimplify and are unwilling to tackle the 
complexities of the real (and messy) world; 

Reductionism is reducing the complex and varied to the simple and 
standard. Its method is to focus on parts instead of wholes. Yet many 
professionals seem driven compulsively to simplify what is complex 
and to standardize what is diverse. Status, promotion, and power 
come less from direct contact with the confusing complexity of people. 
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families, communities, livelihoods, and farming systems, and more 
from isolation which permits safe and sophisticated analysis of 
statistics. It is the reductionist, controlled, simplified, and quantified 
construction that becomes a reality for the isolated professional, not 
that other world out there” (Chambers, 1997). 

The Personal Dimension 

The challenge for managers of those seeking to develop information on 
higher education must be to retain a sense of mission amongst their staff - 
the belief that they can “make a difference” - whilst ensuring that they 
temper this sense with realism. Jowell (1986) in his comprehensive overview 
of the value of codes of professional ethics for statisticians calls the idealism 
“statistical zeal” but warns against the “crusading view of research as an 
instrument of social change” as this leads to many of the most dubious 
ethical decisions. In a multicultural environment, it is even more important 
to be explicit about one’s own motives and values. A transparency of 
interests should be a significant goal, combined with a healthy dose of 
respect for the values and ideas of one’s counterparts. This attitude cannot 
be achieved quickly, as mutual trust and understanding take time to build, 
but they are important investments for future collaboration. 

12. CONCLUSION 

Statistics can serve to benefit society, but, when manipulated politically or 
otherwise, may be used as instruments by the powerful to maintain the 
status quo or even for purposes of oppression. Statisticians working in 
international contexts, usually employed by international, supra-national, 
or bilateral agencies, face a range of problems as they tiy to work with their 
national counterparts to promote policy based on evidence. One of the most 
difficult is the dilemma between open accountability and national 
sovereignty (in relation to what data are collected, the methods used, and 
who is to have access to the results). The meeting in Hiroshima was 
considered as an opportunity for a range of people, who share the common 
goal to improve the quality and effectiveness of higher education systems 
across the world, to debate how we can work together to collect more timely 
and reliable data to inform our decisions. 
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IV. On the Uncertain Consequences of Taking 
Advice from Procrustes 

GUYNEAVE 



1. INTRODUCTION 

The history of the hospitality and hotel trade is long and, some would say, 
glorious. Thanks to the effoils of such heroes as Charles Ritz of Switzerland, 
the Anglo-Italian, Charles Forte, and the American, Howard Johnson, the 
hotel trade has, over the centuries, risen to its present-day cr'owning heights 
in such world -girdling chains as the American Holiday Inn, the French 
Novotel, and the Dutch Golden Tulip. However, for every hero, there ar*e 
many villains unsung, some of whom have attained r'eputations of quite 
mythical pr'oportions. 

The most unspeakable amongst this latter category was a certain 
Procrustes. Procmstes came from an ancient nation for which hospitality 
was a sacred obligation. He was an Ancient and Classical Greek. Nowadays, 
he would find a place in police records as a psychopath with sadistic 
tendencies, much given over to exactitude, precision, and to the noble art of 
measurement, a scholarly rigour that he applied with total impartiality to 
his wretched guests. 

Procrustes, so the legend tells, was the proud owner of a bed. His 
particular bent, his peche mignon, was to ensure that his guests fitted the 
bed exactly and precisely. Guests of restricted gr'owth, he sketched. The 
outsized and oversized he trimmed. Few - if any - survived his attentions. 
He was, if we car*e to think of it in these terms, the pragmatic inventor of the 
notion of “goodness of fit”, well known to statisticians. Had mor'e of those 
engaged in the science of evaluation, and had the elaboration of its 
techniques benefited from a classical education, they would doubtless, have 
accorded Procnrstes the honour of being the unholy father of benchmarking 
as well. 

2. UNTIMELY QUESTIONS 

Frans Kaiser, the Rapporteur for the “Roundtable on System-Level 
Indicator's for Tertiary/ Higher Education”, supplied the participants with a 
list of things to be borne in mind (Kaiser, 2003, in this volume, pp. 31-35), 
to which more could be added. Fir'st, a fundamental question: “What is a 
system indicator'?” “Why do we need it?” These are deceptively simple 
questions. Let me tackle them in r'everse or'der. 

In the first place, one of the most preoccupying developments to light 
upon higher education in the course of the 1990s, even if historians will 
find its origins well before that decade, has been the increasingly 
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foreshortened planning horizon (Delors, 1998), a foreshortening that, in 
many countries, has transferred itself to the evaluative cycle, itself a' 
consequence of the drive for efficiency, publicly stated, and for performance 
and accountability, publicly demonstrated. We see in the notion of the 
“system-level indicator” a lever which, at the very least, will palliate what 
some have termed “the tyranny of the present” and at the very best, restore 
the capacity of the individual university to entertain and act on a long term 
interpretation of its purpose or of its responsibilities - or give it the means 
so to do. 

This author is not sure, however, that taking a symptom of the illness as 
a form of treatment guarantees a cure. We have to ask ourselves why the 
short-term has triumphed, apart from the various forms of new public 
control that are explicit in the Evaluative State (Henkel, 1998; Neave, 1988, 
1998). 

3. THE SPEED TRAP 

It is very much part of the Zeitgeist to believe that the world is a faster place, 
that history - or our rush towards its end (Fukuyama, 1988) - is speeding 
up. It is a perception that lies at the heart of the new theology of today - 
Globalization - and its principal Gospels - Internationalization, 
Regionalization, and the Network Society (van der Wende, 1997; Scott, 
1998; Castells, 2001). Because change - or fashion, which is a very different 
thing - comes upon us faster, we have to move, so this logic argues, in a 
similar precipitation or be thought irresponsible or, worse stiU, impertinent 
- to use a Gallicism current in UNESCO vocabulary. There is, moreover, a 
very explicit threat now going the rounds should we prove so obdurate or 
insensitive to fashion as to dismiss or to ignore it. That threat is contained 
in the specter which the fashionable brandish over the head of the 
university - namely, the hordes of so called “alternative providers”, ready to 
step in to take over the more profitable pails of skill creation and to do well 
by doing good through the expropriation of a public good, namely access to, 
and generation of, information and, more rarely, knowledge. 

It is arguable - and many have done so (Scott, 1998) - that the situation 
the university faces today has no precedent in the eight hundred or more 
years that constitute the history of the European edition of higher learning. 
The implicit argument, which follows from this point, is, of course, that 
there is nothing to learn from history and, indeed, history is both powerless 
and, last, in contemporary insults, impertinent. We can cheerfully forget it, 
however, whether or not our grandchildren will be as cheerful about the 
consequences of the history we shall certainly make, is perhaps best not 
even considered. Let me point out one thing, however. The university has 
not always been conceived as an institution, the obligation of which it is to 
respond in panic-stricken fluttering in the face of the here and now. It was, 
on the contrary, the agent of long-term change and general modernization 
(Neave and van Vught, 1994). 
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From this perspective, our task to identify system indicators may be seen 
as an effort to return the university to this engagement and to furnish an 
instrumentality for diminishing the ephemeral. It might even provide us 
with the lever that would permit the sloughing off of those elements of 
training which, by their nature, have to obey the imperative of the short 
term and which may well be better sited in the arms - or the cash registers 
- of the “alternate providers” (Nybom, 2000). 

Such heresy may not be pleasing. But it raises the question of whether 
the system-level indicators for which we are groping are to be universal 
across all institutions of higher education or whether they are not more 
suitable for some of them, just as tactical indicators - supposing we reach 
an operational distinction between the two - are certainly more pertinent for 
other institutional forms in a system the variety and complexity of which 
grows greater by the year. 

4. ANOTHER ESSENTIAL ISSUE 

To turn to the first of the two questions posed earlier - that is - to the notion 
of a “system indicator”. Is there such a thing as an indicator that, by its 
innermost essence, is truly “systemic” as opposed to indicators that are 
used or which serve a systemic purpose? One of the assumptions of this 
Roundtable was that we should look to our instruments, to our tools. It was 
presumed that if we can find pointers that allow us to concentrate the 
attention of institutions upon the basic frame factors that shape their long 
term purpose and which give them a good chance of attaining it, we shall 
have done well by the Prince, by the Republic of Scholars, and by the 
University Administrators of the world. However, there is another side to our 
basic mission. The other side requires that we not ignore entirely that 
equally important fact of how indicators are used. We have to be just as 
sensitive to the honest artisans, technicians, and latter-day sons of 
Procrustes who, like their ancestor, use them or misuse them. 

5. MEASURES OR MEN? 

Behind our exercise in exploring this domain, we come back to a very basic 
issue which, these ten years or more, has run beneath much of the reforms 
; in Western Europe and elsewhere as well; namely, what is the right balance 
in achieving system-level goals between instruments and measures versus 
men and women. Over the past decade, we have inclined most decidedly 
towards the view that instruments are the prime forces, which is an 
alternative view upon the assumptions that underpin the Evaluative State. 
Still, there is another school of thought which, following the paradigm of the 
business firm, argues that leadership is no less important (Hirsch and 
Weber, 2001). Our Roundtable would appear to have fallen into the former 
approach. It is for this reason that 1 underline the latter thesis. 

Let us also remind ourselves that indicators are precisely that. They 
point towards some item or process. They are very often surrogates - that is. 
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replacements for activities, which, sometimes, do not easily lend themselves 
to quantification (Johnes and Taylor, 1990), but are held to be most useful . 
even so. Let us also bear in mind that indicators are, perhaps, the supreme 
example of two processes - abstraction and de-contextualization. They serve 
the purpose of extracting what is held to be information necessary for 
governments to assign resources, to induce modifications to institutional 
behaviour, and to inform the public. They provide information. They 
indicate difference or similarity on a stipulated dimension or criterion. They 
do not necessarily provide either knowledge or understanding. In other 
words, they do not necessarily provide an explanation for whatever degrees 
of difference or similarity that they may throw up between systems. For 
that, one must seek elsewhere. Last, but very far from being least, they 
represent only one account of whatever unit is measured. 

6. THE RETURN OF CONTEXT IN POLICY-MAKING 

One of the most important developments in the application of public policy 
that has taken place in the course of the past decade lies precisely in the 
opposite direction - that is, the return to context as a fundamental part of 
allocating resources. Indeed, the importance of context is so significant that 
it has permeated even the spheres of world banking (The World Bank, 
2000). Such a rediscovery of context is of the utmost significance for at least 
two reasons, both of which have bearing on our task. First, it presumes that 
single mode accounts of higher education, however necessary they might 
be, are patently not sufficient - even for policy-making and above all not at 
the highest levels of aggregation, effectively the system-level. Second, it may 
be argued that the return of context bids fair to undermine the relevance of 
universal “package” solutions of which de -contextualized technique 
appeared once to legitimate and justify within the setting of the nation- 
state. Not untypical of this earlier, indicator-driven approach to higher 
education policy was that previous phase in World Bank activity which 
revolved around “rates of return” analysis (Psacharopoulos, 1988), often 
with the heartiest disregard for the political, social, and institutional 
environment into which they were injected. 

On their own, “de-contextualized solutions” are an interesting 
phenomenon, a notion of homogeneity which once underpinned the higher 
education systems of many nation-states (Neave and Van Vught, 1991; 
1994), but which today appears to be migrating beyond them to take refuge 
as a fundamental buttress to the theses of globalization and 
internationalization. Like Procrustes, we find it a useful perspective through 
which to puU our foreshortened systems of higher education to fit a larger 
world. 

7. THE AMBIGUITY OF SYSTEMIC INDICATORS 

What makes an indicator “systemic”, 1 would suggest, does not lie primarily 
in the indicator itself. Our ingenuity may well come up with some. 
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Nevertheless, what makes an indicator systemic is not simply that it focuses 
on the systems level. It is also systemic because it invigorates the 
imagination of those who have recourse to such an instrumentality to 
inform their judgment and to enlighten their paths, which their vision often 
and their colleagues, sometimes, have revealed to them. Nor do I exclude 
the possibility that some might be encouraged to take a broader view by the 
force of persuasion that indicators often have. Whether such compliance, 
however, will bring about “deep change” is an entirely different matter, as 
recent studies in Swedish universities (Bauer et al., 2000) and certain 
suspicions amongst the Dutch Higher Education Inspectorate (Scheele et 
ol., 1998) seem to hint. 

We should exercise judicious caution as to the consequences of what we 
do. The adding of further indicators, if they are taken up and acted upon by 
governments, can just as well work in the opposite direction. Our 
Rapporteur warns us against the ambiguities of interpretation, and he is 
right to do so. For whilst they may assist and stimulate imagination and 
daring, they may equally serve to paralyze, if not overwhelm both virtues, 
when they are weak. Moreover, indicators can never act as a substitute for 
the absence of vision, nor for one that is faulty in the first place. 

Certainly, they may help us in knowing, for instance, how far we are - or, 
for that matter, how near we are - along the route to achieving our elected 
path, and very useful they are in this respect. For indicators to be sensitive 
or even, for that matter, to be adequate, presumes a very high degree of 
consensus about the goals a higher education system wishes for itself or is 
instructed to meet - whether by governments or by international trading 
organizations. What is also required, to no less a degree, is the ability to 
take account of differences in models of development which individual 
systems incorporate or represent. This task is no small one because the 
basic assumption behind the use of individual system-level indicators is 
that the weighting of what they measure and its importance within the 
national agenda of higher education policy at the systems level is similar for 
all systems measured. This assumption, to say the least, is one of an 
exceedingly tall order. 

8. TOUT CE QUI BRILLE N’EST PAS DE L'OR 

All that glistens is not gold, and not all those things about which we would 
like to know more are measurable, even though, certainly, we can choose 
those which are. However, here again what can be measured is not 
necessarily the same as what ought to be measured. Furthermore, the price 
of interchanging quality and quantity tends to be very great, just as the 
price of forcing the traveler into Procrustes’ king-sized four-poster was very 
great for the miserable traveler. The price of locking in the imagination of 
leadership, administration, and academe, of seeking to confine them within 
the often unspoken assumptions that stand behind the individual 



“All that glitters is not gold" (William Shakespeare, The Merchant of Venice, Act 2, Scene 7). 
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component measured can often be - as the British have found to their cost - 
high in cash committed and very high indeed in terms of erosions in trust, 
confidence, and good will among higher education, government, and the' 
public (Trow, 1998). 

Yet, it would be foolish to take the view that indicators are devoid of any 
virtue. At a time when higher education is becoming in its myriad forms, 
varieties, and variations, so complex as to beggar the analytic abilities of the 
best, indicators do at least cut through that complexity. Whether in cutting 
through, they yield a true - as opposed to an illusory - transparency, an 
artifact of their own apparent precision, is a very different kettle of fish. 

9. DIFFERENT ACCOUNTS - SUPPLEMENTARY AND COMPLEMENTARY 

There are many different accounts of higher education. Indeed, their 
number is increasing aU the time. From the twenty- two or so perspectives 
Tony Becher identified at the start of the 1990s (Becher, 1998), others have 
come to join them in the course of that same period. Mobility studies, 
international relations in the sense of student and staff exchanges between 
countries and across continents, quality studies, and evaluation studies are 
not the least of the fields which have burgeoned in the intervening period. 
Our knowledge of higher education is becoming richer. Whether it is not too 
rich for those who, like governments and consumers, seek information 
about where resources both public and private should be invested, is a 
question not lightly answered. 

Even so, indicators - above all, system-level indicators - need to be 
complemented by other accounts if only for the fact that as a “stand-alone” 
mode of trying to gauge what is happening, they may inform, but they have 
little or no power to explain. They reveal differences. They may have a 
devastating plausibility. League tables are a particularly nice example. But, 
they do not tell us how to account for those differences - hence the need for 
other accounts which advance our understanding and that give us 
understanding of whatever trends stand revealed. 

Plausibility is very far from being the same thing as understanding. And 
so, for that matter, is information. Moreover, this fundamental and inherent 
weakness in system-level indicators is compounded further. Difficulties in 
achieving understanding grow as the numbers of self-standing systems 
brought into to the realm of measurement grow, whether they are regional, 
national, or continental. Indicators are created in a context. Whilst 
multiplying different contexts may yield a succinct, apparently plausible, 
and an outwardly pleasing comparable tableau, it may, as one eminent 
British civil servant explained a few years ago to an Australian Commission 
of Inquiry, also be “somewhat economical with the truth”. 

10. DOXOLOGICAL DRIFT: A MORTAL SIN? 

There is, however, a further phenomenon that tends to accompany the use 
of indicators in higher education. Whilst it is most evident in systems in 
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East and Central Europe, it is no less present in Western Europe. The term 
coined to describe it, however, owes more to religion and belief than to the 
exact science of measurement. It is “doxological drift” (Tomusk, 2000). 

Let me explain. A doxology is a sacred text - the source for the beliefs 
and revealed wisdom to which the Faithful, if they are not to be considered 
heretics, must subscribe and to which, in justifying their arguments and 
actions, they must inevitably refer. Some religions have many sacred texts. 
They are, accordingly, heterodox in the nature of their sacred knowledge. 
Others, by contrast, are characterized by the fact that their sacred 
knowledge is contained in a single text. Accordingly, they subscribe to an 
“orthodoxy” - to a single wisdom, often represented by a single 
interpretation and a single institutional form. There is, in short, “no 
alternative” - a modem and secular phrase which prolongs a very old 
mentality! For those with an historical turn of mind, the old Latin tag may 
not be entirely forgotten: Extra Ecclesiae, nulla solus - “Outside the Church, 
there is no Salvation.” 

Although relatively little is known at present about the exact 
mechanisms contributing to “doxological drift”, it has both an important 
and highly pernicious, not to say perverse, effect. Doxological drift in the 
form it takes in higher education involves a displacement in the function 
which authority lays upon indicators. In theological terms, indicators move 
from fulfilling a heterodox purpose and take on the role of ensuring - and 
enforcing - “orthodoxy”, that is, they are held to uphold one interpretation 
alone. Put in plain English, indicators become stripped of their 
interpretational ambiguity. They are also stripped of their tentative purpose. 
They mutate into hard and fast criteria of judgment, assessment, and 
evaluation. In terms of curricular theory, they loose their formative function 
and take on a summative purpose. 

In short, an instmmentaliiy designed to provide a provisional statement 
about the positioning of various national systems of higher education, 
mutates into a framework of control by which systems, and their 
performance are judged, weighed in the balance, placed in a rank order of 
glory and ignominy - or all three. Thus, an instmmentaliiy which, in its 
pristine state, pointed up and highlighted variation, becomes part of a 
broader procedure, the one tme path to righteousness, an instmment of 
close control, scmtiny, and much vexation. 

1 1. A DISSENTING VIEW 

But “doxological drift” has nothing to do with indicators per se. It is rather a 
consequence of attributing more power to indicators than they can bear or 
even justify. It is, in effect, a distortion of the principle of indicators which, 
as mere techniques of measurement and ascertainment, are as innocent - 
and as neutral - as the driven snow. 

“Every indicator pleases 

And only Man is vUe....” 
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to parody the poetic efforts of Reginald Herder, an early Nineteenth Century 
Anglican divine. 

Yet, “doxological drift” is far too general a phenomenon - present as 
much in the systems of East and Central Europe (Tomusk, 2000), as it is in 
certain unfortunate islands in the West - for us to dismiss it when we pick 
and choose and draw up our list. Certainly, we can readily appreciate why it 
comes about, if not always how. It comes about precisely because the 
information with which indicators supply us needs to be interpreted and 
understood, to be, if you wiU, reinserted back into what some are pleased to 
call “the policy context”. And this necessity, as mentioned earlier, merely 
underlines once again the importance of context. Indicators, above all at 
system-level, alter the context of higher education just as their own pristine 
purpose may be altered by their contexts - regardless of the particular focus 
taken on it and regardless of whether that focus is construed and analyzed 
in terms of “the market”, as official ideology, or as bureaucratic politics. 

Whether we can, by anticipating this apparently unavoidable 
phenomenon, protect ourselves against it, is a delicate matter indeed. There 
is, however, a further development, and it too is no less Procrustean. 

12. ANOTHER FORM OF DIFFERENTIATION AND DIFFERENCE? 

Because indicators are deemed to be the trusty servant of transparency - 
indeed a veiy great part of their power resides precisely in their availability 
to the public - measures intended to shed light on one aspect of system 
behaviour often take on a different purpose when used by different 
constituencies with veiy different interests. This too is understandable given 
the often-astonishing opacity which, until recently, cut off what higher 
education had achieved - and how - from aU save the most persistent 
members of the public. Certainly, some information is better than none. But 
knowledge that has meaning and validity is better even still. 

It is at this point that we ought to permit ourselves the luxury of asking 
whether or not system indicators should overridingly be linked to outcomes. 
Nobody in his or her right mind wiU question the need for some - and 
amongst them the most crucial - to rest precisely on what a system of 
higher education or a segment within it, has achieved. They are 
indispensable for governments, for taxpayers, for industry, and not least for 
parents, all of whom have choices to make in investing resources and who, 
in their var 5 dng ways, regard outcomes as plausible demonstrations of 
excellence and relative efficiency. One may, however, ask whether or not 
outcome based indicators, and graduation rates in particular, are always 
useful for prospective students. 

13. MODES OF INDICATORS AND THE STUDENT ESTATE 

In a world that, we are told, will be one of unremitting change and spreading 
diversity - in institutional form, mission, mode of study, duration, and in the 
sheer variety of programmes available within the higher education system of 
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a country - it is also logical that the mass of students evolve similarly. And it 
is doing so. Indeed, in certain mature systems of higher education, it is not 
misplaced to speak of the “fragmentation” of the student mass into age 
groups, occupations, types of studies undertaken, etc. 

Against this backdrop, it seems paradoxical in the extreme that so 
diverse a constituency can gain even a modicum of information about the 
provision, programmes, and courses on offer when all they are provided is 
often presented in terms of the performance of alumni about whom little is 
known that is up to date at system-level, save that they are alumni. In fine, 
our indicators, which purport to inform future students about the quality of 
provision, tend to rest on the implicit assumption that they will perform in a 
similar way to their erstwhile fellows. If this is not determinism, it is most 
certainly faith and hope in the absence of charity. Faced with a 
heterogeneous student body, what is needed are indicators about the 
quality of provision - in effect, another account to give meaning to quality as 
measured by outcome. Measures of student outcome, of graduation rates, 
may be satisfactory when higher education is cast in a “productivist” mould 
which, incidentally, outcome indicators reinforce. But they are unlikely to 
constitute direct knowledge of the quality of the provision of a given nation 
in different disciplines or domains of learning. In effect, we need to consider 
system indicators that go beyond outcomes and which allow students to 
have direct knowledge of the quality of what national systems may offer 
them, rather than relying on unsystematized - and unverified - publicity on 
the internet or on deductions as to quality and excellence based on the 
achievements of their elders who are not necessarily their betters. 

14. CONCLUSION 

In this article, the author has deliberately eschewed committing himself to 
recommending particular systems indicators. Such is the work of particular 
specialists. Rather, the concern of this author focuses on the consequences 
that the strengthening of one form of information may have upon the ways 
we view £md seek to understand the complex workings of the higher 
education system. I have to confess to a certain misgiving about this 
exercise, not because I think it is not necessary. The more structured and 
regular information we have on the development of the systems of higher 
education in the world - in all their variety and difference - the greater the 
opportunity to test and to develop alternative accounts that transform such 
information into knowledge. I have to confess, however, to a certain 
trepidation at the temptation to see the technique of indicators as the most 
succinct way of understanding what is going on in the higher learning 
system, and because it is succinct in a world of haste, therefore the most 
weighty, the most appropriate, and the most influential. 

The trend towards “orthodoxy” - of one right and desirable model, 
irrespective of how successful its original template appears to be - is rarely, 
if ever, fruitful for either creativity, scholarship, and maybe, not even for 
democracy. But that is a different agenda. And yet, by suggesting that we 
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should elaborate a series of world-girdling indicators at system-level, we 
should be wary indeed of the path down which such a course may lead us. 
Orthodoxy, like the self-fulfilling prophecy, is self-sustaining and, more to 
the point, self-validating. Naturally, in choosing certain indicators, we shall 
do so because we believe in - and, just as importantly, have evidence for - 
their utilitarian value, accuracy, and sensitivity. Yet, we have to be very 
clear on one thing: we are seeking to pinpoint developments that we hold to 
be important at this precise moment. That does not mean to say that other 
aspects will not assume equal or even greater importance later or that their 
impact is not beginning to accumulate. Of these we should be aware. They 
may require our attention later. Indicators have then a temporal context 
and for that reason, they are also provisional. 

There is another side to avoiding “doxological drift”. When we set a 
universal set of criteria to map the current condition of higher learning, we 
are also setting higher learning within a single overall frame of reference 
and also, of expectation and purpose. If, to the artisans of measurement, 
indicators are instruments that let us know what is happening, to 
governments and those who shape the way of the world, they are pointers to 
a purpose, sometimes achieved, often in the process of implementation or, 
rarely but not unknown, facing dismal failure. They are then instruments of 
intent, and intent is policy by other means, however, whether such policies 
are outwardly avowed or dare speak their name is not always as clear as 
many might wish. 

System indicators are one level below global indicators and, of course, 
they can serve to track that selfsame process with the greatest of ease and 
very little adaptation at all. There are many models of globalization, some of 
them based on antithetical and conflicting ethical and moral principles - 
competition versus co-operation (van der Wende, 2001), expropriation 
versus restitution, to mention but the most obvious. To ask which of these 
masters we are to serve is indelicate. But not to pose such a question is 
unpardonable. Procrustes’ guests had at least the right to ask the price of 
the lodging, regardless of what happened to them afterwards! 
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1. THE NEED FOR INDICATORS 

In many countries there are university rectors, presidents, or vice- 
chancellors who have little quantitative knowledge of what is actually 
happening in their institutions. In the name of academic freedom, they 
confuse autonomy with total independence. In these countries, the notion 
of national accountability is viewed as outlandish. 

In many countries, there is no ministry of higher education and no 
government authority responsible for higher education. Some Latin 
American countries have a Council of Rectors, but the universities are 
autonomous and do not report to anyone in actual fact. A significant 
share of the higher education system escapes the authority of the 
government in countries in which 60 to 70 percent of students attend 
private universities. 

In other countries, such as Chile, in addition to the public /private 
sector divide, a distinction between old and new universities is made. 

And then, in many countries, one of the reasons for which the World 
Bank deals with indicators is that universities have budgetary 
entitlements. In Central America, for example, the universities receive 6 
percent of the national budget, whatever happens. Therefore, why should 
these universities worry about indicators? 

Why is information needed? For diagnostic purposes: if one wants to 
look at a higher education system and to find out what is wrong, what is 
right, and what kind of accountability is given. If one accepts that the 
budget represents public money, then there is the question of responsible 
accountability. And, finally, there is a point about operating in a 
competitive envir'onment, where it might be useful to know what one’s 
: competitors are doing and how well. 

For 150 years, the venerable Universidad de la Repdblica [University of 
the Republic], the oldest university in Uruguay, was in a monopoly 
situation. It represented the entire higher education system of the 
country. There was a very collegial relation, with everybody important in 
the country being a graduate of that university and voting for the election 
of the Rector, from the President of the Republic to all professors, 
janitors, students, and former students. No change had ever occurred in 
that university. And then, in 1986, it experienced a wake-up call with the 
establishment of the first Catholic University in the country. 
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Because Uruguay itself is a good Roman Catholic country, the people 
at the University of the Republic could not do much about the setting up. 
of this new university. But then other people told themselves, “If we can 
have one private university, why not more?” So a few entrepreneurs 
established more private universities. Then, the public university went to 
court. It went all the way to the Supreme Court to try to prevent these 
private universities from being established, but the Court said yes, there 
could be competition in higher education. And for the first time in the 
history of the country, the University of Umguay started to benchmark 
itself a little bit against its competitors and decided to establish Master’s 
Degree programmes, for there were no Master’s Degree programmes in 
Uruguay until the Catholic University established the first one. 

Finally, a very important reason for looking at indicators is to make 
informed choices about the future. The Physics Nobel Prize winner, Niels 
Bohr, once said, “Forecasts are typically difficult to make, especially 
about the future”. Nevertheless, it is important to try to think forward, 
accepting that one will not achieve exactly what one wants but, at least, if 
one has the baseline, one knows where one has started, and then one can 
plan ahead and take stock. 

Who needs what type of information? Is there a set of indicators for all 
situations, applicable to all universities, in all countries? Or do we need 
different types of indicators, depending upon the question asked? 

Who are the beneficiaries of the information provided by indicators? 
First, the state, for purposes of accountability, then the public at large, 
which needs to know where the system is going and how well individual 
institutions are performing. Then come the institutions themselves. And 
finally, the donors need to make sure that the resources, which are 
transferred to the recipient countries, are well used. 

2. THE WORLD BANK EXPERIENCE 

The first instance in which the World Bank makes use of indicators in the 
domain of higher education is in what is called “analytical work”. The 
World Bank works together with countries to make a diagnosis of the 
most salient issues in higher education. To do that it has to design 
system-wide and institutional indicators. 

Second, one of the missions of the World Bank is to provide quality 
technical advice to countries and institutions. It helps them look at 
different reform options. 

There are three major forces of change interacting in higher education. 
The first one has to do with the main dimensions of regulation coming 
from the State. The second one represents the participation of civil society 
and partnerships with employers. The third one shows the competitive 
environment in which higher education institutions operate. At the 
intersection of these three sets of forces, there are critical factors, such as 
quality assurance, ranking systems, formula funding, etc., mechanisms, 
which can generate useful sets of indicators. Depending upon the aspect 
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of the system upon which one wishes to focus, one can choose a varying 
set of indicators. 

Developing indicators at the institutional level is equally important. A 
few years ago, the Rector of the Catholic University of Lima, Peru, needed 
help in starting a strategic planning exercise, for over the years, the 
University had been losing students. The University is located in what 
used to be the main area of the city, but with urban growth, there had 
been significant population shifts. Today, the university finds itself in a 
district that is very much one of low-income. One of the hypotheses to 
explain the loss of students was that students from middle- and high- 
income families were living in a different part of the city. 

The University developed a set of indicators and conducted surveys 
among students and employers. They found out that the real problem 
was not geographical, but rather that some departments were perceived 
as being of poor quality. 

Finally, indicators can be very useful tools for making international 
comparisons and for engaging in benchmarking across countries. The 
World Bank has just organized two stakeholder consultation workshops 
in Sri Lanka and Bangladesh and some cross-country comparisons were 
used to compare the realities in several South Asian countries. 

Also, in some of the World Bank projects, the establishment of 
Management Information Systems (MIS) is financed. These systems serve 
as the basis for the production of system-wide and institutional 
indicators. Management Information Systems are being created, for 
example, in Argentina and Bolivia. In Argentina, the World Bank is 
supporting the network of public universities, helping it set up 
management information systems and establishing Intranet/ Internet 
connections. 

In Bolivia, the Dutch Government has been providing technical 
assistance to the second university of the country, the University of 
Cochabamba. This assistance was undertaken in a very co-operative 
manner, so much so that, at the end of the process, the Dutch university, 
which had been assisting the project, the Free University of Amsterdam, 
decided to transform its own Management Information System, back in 
Amsterdam, with the help of their Bolivian partners. And now, as a 
. second phase, the World Bank is helping the Bolivian University to form a 
partnership with other public and private universities to implement the 
same management information system reform with them. 

The World Bank has a project to support a new Student Loan Agency 
in Mexico for students in the private universities. The latter developed 
their own Agency. The federal Government took a loan from the World 
Bank and lent it on to the private Agency. In the course of project 
preparation, the World Bank developed a series of indicators: demand 
and targeting indicators, financial indicators, and institutional operation 
indicators. The main purpose of this exercise was to formulate, along with 
World Bank counterparts, relevant measuring instruments to look at the 
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impact of the project, to be able, at the end of the project, to see whether 
its objectives had been attained. 

3. WHAT LESSONS HAVE BEEN LEARNED AS PROJECTS WERE 
DEVELOPED? 

First, it is important to stress that dealing with indicators is not only a 
technical activity. There are important political dimensions involved as 
well. For example, in Argentina, at the beginning, the universities were 
very reluctant to participate, for they viewed the project as an 
imperialistic effort by the Ministry of Higher Education to reduce their 
autonomy. 

So the World Bank decided that, instead of having a set of experts 
working at the national level in the Ministry, designing the ideal modules 
of indicators, the universities themselves would be involved in the 
preparation of the management information system. The project team 
identified a university that had a good module for academic management. 
Then, the universities were asked whether they would be willing to share 
this module with other universities. When they agreed, they took the lead 
in developing and adapting that particular module. 

A second important point is that indicators should not be perceived as 
threatening. For instance, if the announcement is made that performance 
indicators will be linked to funding, a negative message is being sent. In 
this context, the interesting experience of the Provdo examination in 
Brazil is worth mentioning. The Provdo is an aptitude test administered to 
undergraduate students three months before they complete their studies. 
The results of the test do not corint towards their graduation, but they 
are used to measure institutional performance and to make comparisons 
between public and private universities. It is interesting to note that even 
though it is a voluntary test, employers increasingly ask Job applicants, 
“By the way, what score did you get on the Provdo?” 

A third dimension is the need to make good use of these indicators for 
purposes of planning, management, and monitoring. Producing them is 
one thing, but, if they are not put to use, much effort and resources have 
been wasted. A management information system is only the first step. 
Even more important is to develop a culture of application of these 
indicators. In Colombia, for example, the results of the accreditation 
process are published widely, and students who enroll in accredited 
programmes are placed on the priority list for receiving student loans. 

4. CONCLUSION 

Constructing indicators can be a very useful and powerful exercise that 
can have, beyond the technical dimension, some important political 
implications. After the reunification of Germany after 1989, a team of 
professors from West Germany was placed in charge of evaluating the 
East German higher education system. Since there was no tradition of 
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evaluation in West Germany, the professors had to invent and deal with a 
new set of indicators. At the end of the exercise, some of the professors 
asked whether it might not be useful to undertake the same exercise in 
West Germany. The process of asking ourselves what we want to measure 
and for what purpose can be as fruitful as the result. 
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VI. The German Perspective Regarding the Design 
and Use of System-Level Indicators for Higher 
Education 

KLAUS SCHNrrZER 



1. SYSTEM-LEVEL INDICATORS AND THE SHIFT OF PARADIGMS 

Reports on system-level indicators for higher education in Germany tend to 
yield a more or less similar catalogue of performance indicators as is the 
case in most of the countries bordering Germany. A very good systematic 
description of system-level indicators for higher education in Germany was 
produced by Karl Alwell in the OECD-IMHE compendium, The Development 
of Perforrrumce Indicators for Higher Education [Alwell, 1993, p. 61). It listed 
types of indicators and sectors of application. 

This study will not repeat that exercise. Rather, it will attempt to analyze 
the strategic impact of these indicators for higher education in Germany at 
national level and in top-down or bottom-up processes at institutional and 
national level. 

As stated in the outline of the UNESCO Project, “Strategic Indicators for 
Higher Education in the Twenty-First Century”, factual reporting on 
indicators should not be a goal in itself. The main concern should be to 
work out the political context in which these indicators are to be used and 
how they gain or lose importance as political issues change over time. 

Only in this context-relationship can performance indicators reveal their 
strategic meaning and strength and answer some of the “self-challenge 
questions to perfonnance indicators” posed to national experts by Karen 
Abercromby and John Fielden, in their study of higher education indicators 
( 2000 ). 

The present study attempts to answer part of the question as to how 
German indicators match the vision of the World Declaration in regard to 
performance indicators (UNESCO, 1998). Its overall goal is not easy to 
achieve, for in terms of the German context of higher education policy in 
which performance indicators are embedded, a clear line of policy issues 
and trends in the state governance of higher education is difficult to trace. 
This difficulty must be explained as it makes clear that there are sets of 
indicators which do not suit the same rationale and which are interpreted in 
different ways according to the views of different stakeholders. 

1.1. New and Old Public Management 

The past decade in Germany was characterized by a wave of reforms in 
higher education aiming at the introduction of market mechanisms at 
national and institutional level. This process of change is still going on. 
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There are hardly any articles on education and conferences on higher 
education issues in Germany that have not discussed aspects of autonomy, 
competition, profile shaping, benchmarking, ranking, de -regulation, 
performance oriented funding, etc., on the one hand, and recommendations 
as to how to overcome obsolete State-regulated control and provision 
mechanisms, on the other. The discussion is dominated by the new 
paradigm of a market regulated higher education system. 

This new paradigm not only develops new forms of strategic governance 
but also, as a consequence, a new set of indicators. Thus, it seems logical to 
concentrate the following paragraphs on the new role of indicators in the 
market-oriented higher education system of Germany. But as happens very 
often with fashion, the fitting of new clothing may catch the eye but 
continues to cover the prevailing functions of the body. The same seems, as 
well, to be true with regard to the new management concepts and steering 
indicators of the higher education system in Germany. Even though “new 
public management” issues are dominating discussions in the 
scientific /academic community and a debate on different market 
approaches is occupying the new “controlling” professions, one cannot deny 
that the higher education system in Germany is still fundamentally 
organized according to State regulations and planning indicators. 

1.2. Conflict or Convergence of Paradigms 

Attention has to be drawn to these conflicting paradigms as it is quite 
obvious that neither the wishful thinking of the traditionalists - fashions 
will change by themselves - nor the missionaiy optimism of the market- 
supporters - that market forces will supplant the concept of the welfare 
provision of study places will come true. 

From a distance, the two conflicting concepts can be reduced to two 
positive goals. The goal of enhancing the quality of the higher education 
system in Germany calls, indeed seriously, for the introduction of 
competitive market elements into the steering process, while, on the other 
hand, the constitutional provision of free access to equivalent study 
opportunities calls for protective instruments for assuring, at the same 
time, the dominance of social demand. 

1.3. The Third Way 

As both goals find justification in the higher education system of Germany, 
they must be taken into account, now and in the long run, when systems of 
indicators are discussed. The question of if and how these too seemingly 
contradictory paradigms can be reconciled is an open one as, also, is that of 
whether the necessaiy convergence of the two concepts will suggest a “third 
way” of study reform, one suitable for Germany that might also be applied 
in other European, formerly purely state -regulated, higher education 
systems. 

Christoph Oehler, a Nestor among German researchers on higher 
education, recently argued dialectically, in a provocative article (Oehler, 
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2001, p. 28) titled “Geplant wird immer noch” [There Is still Planning], that 
the basic consensus on the welfare state function of the higher education 
system can best be sustained by bringing together the system-level 
planning competencies that steer the higher education system and the 
institutional-level market mechanisms so as to optimize matching processes 
between the national planning framework and Idle institutional setting. By 
this means, universities and other institutions of higher education might 
better participate in the overall responsibilities of the State and of State 
planning. 

At the moment, virulent conflicts and counterproductive antagonisms 
are more the rule than productive interaction between these two competing 
issues. This situation is made very obvious if one observes that system-level 
indicators are in parallel use in Germany. If the UNESCO project, “Strategic 
Indicators for Higher Education in the Twenty-First Century”, could lead to 
the reconciliation of the two sets of indicators and their use, the result 
would be a step forward in overcoming the momentous stalemate in which 
the German debate is locked. It would contribute to the design of an 
indicator system of the “third generation”. With this perspective in mind, the 
next two sections of this study lay the foundations of a discussion by 
demonstrating the parallel uses of indicators, first in a planning setting and 
second in a performance setting, in the actual higher education system in 
Germany. 

2. KENNZAHLEN AND RICHTWERTE - SYSTEM-LEVEL INDICATORS FOR 
HIGHER EDUCATION PLANNING 

The common indicators used in reform phases through the 1980s were key 
numbers (Kennzahlen} and standards [Richtwerte], The naming of the terms 
reveals the specific use of system-level indicators in the restructuring and 
consolidation period from 1945 to 1960 and the succeeding phase of active 
higher education development and reform policy from around 1960 
onwards. 

The West German higher education institutions were placed under 
considerable pressure to change and to modernize, which, in turn, initiated 
a phase of fundamental reform in the higher education sector. This phase began 
in the first half of the 1960s, reaching its climax in the mid-1970s, with the 
passing of the Higher Education Framework Act (Hochschukohmengesetz - HRQ 
in 1976, and ended shortly thereafter. 

The most influential factor in determining higher education development 
in West Germany between 1960 and 1980 (and beyond this period) was the 
continuous expansion in the demand for study places, a process which is 
still underway today. 

The responsibilities of the Federal Government with regard to the 
funding and organization of the higher education system and scientific 
research have been extended since the end of the 1950s. By 1956, the 
Federal Government had started to participate in higher education funding, 
particularly in the areas of the expansion of higher education institutions 
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and the promotion of research and financial support for students. This co- 
operation between the Federal Government and the Lander was put on a ' 
new constitutional footing at the end of the 1960s (Amendment to the 
German Basic Law [Grundgesetz - GG] of 1969). 

On the basis of these provisions, the following Federal laws were enacted: 

- the 1969 Hochschulbauforderungsgesetz [HBFG) regulating the planning 
and funding of the expansion and construction of higher education 
institutions: 

- the 1971 Bundesausbildungsfdrdemngsgesetz [BAfoG) regulating 
student financial support: 

- the 1976 Hochschulrahmengesetz [HRG) regulating the structure and 
organization of the higher education system. 

The consequences have been as follows; 

- the expansion of the existing higher education institutions and the 
construction of new ones. The rapid growth in the numbers of 
students since 1960 has also led to the expansion of existing higher 
education institutions (personnel, premises, and courses), as well as 
to the founding of a large number of new higher education 
institutions. 

- admission restrictions and “overload”. The rapid growth in the 
numbers of first-year students and, indeed, students as a whole since 
the second half of the 1960s in certain subject areas, especially 
medicine, has led to serious bottlenecks and capacity problems. 
Despite the increased expansion of higher education institutions, 
admission restrictions (the so-called numerus clausus) have been 
introduced for a range of course programmes that are in particular 
demand. In 1972, following a ruling by the Federal Constitutional 
Court, which set narrow constitutional limitations regarding the 
introduction of admission restrictions, the Lander signed a convention 
regulating selection and admission procedures and set up a central 
body for the allocation of study places [ZVS] in Dortmund. It is 
responsible for the administrative organization of the selection 
procedure for a limited range of course programmes. The introduction 
of admissions restrictions represented - in historical terms - a first 
break with German higher education tradition, as the higher 
education institutions in Germany had always been relatively open 
and accessible. 

- institutionalization of higher education planning. As in other areas of 
public administration, planning concepts and procedures, sipce 1965, 
have gradually become part of the higher education development 
process in the Federal Republic of Germany. However, only during the 
heyday of higher education reform policy was long-term planning, te., 
the concept of systematically steering the future development of 
higher education by means of longer-term programmes, pursued. 
These plans defined the relevant objectives, the measures planned for 
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achieving them, and the corresponding implementation stages. State 
higher education planning changed fundamentally as early as the 
second half of the 1970s, when longer-term target-oriented overall 
planning was replaced by an administrative approach seeking a 
solution to acute capacity shortages. 

- expansion of State steering of the higher education system. There has 
been a marked change in the role played by the State in higher 
education since the end of the 1950s. The State no longer limits itself 
to its traditional role, concentrating primarily on guaranteeing the 
legal and financial framework for the continuance of otherwise largely 
independent (“free”) research and teaching activities. 

The reasons for the increasing influence of the State lie, on the one hand, 
in its expanding financial and planning involvement in the higher education 
sector, as well as, on the other hand, in the widespread impression that the 
higher education institutions themselves did not have the strength to 
initiate the required reforms by themselves. 

Fundamentally, a distinction can be drawn between three types of State 
steering of the higher education system; 

i) statutory standardization in the Lander and at federal level, 
culminating in the passing of the Higher Education Framework Act 
(Hochschulrahmengesetz ) ; 

ii) intensification of State steering through decrees and administrative 
regulations below the statutory level; 

ii) a growing number of judicial rulings (by the constitutional, 
administrative, and law courts) which led to extensive regulation and 
bureaucratization of more than the internal situation. 

This legislation for restricted expansion needed incremental planning 
tools, among others, Kennzahlen and Richtwerte. Most of them were 
concerned with planning and equal provision of resources. 

The provisions concerning the award of study places, that mainly cite the 
responsibility of the State to offer as many study places as needed, refer to 
the human right to free access to education. The procedure for awarding 
study places was and still is the main field of operation of Kennzahlen and 
Richtwerte in Germany (see below). 

Another important field of action was the design and use of social 
indicators to measure the promotion of social demand itself. As mentioned 
above, apart from the legislation to guarantee the supply of study places, a 
legal framework was introduced to support and raise social demand; 

The so-called BcfoG Law 

- offered individual support to all German students: 

- ensured equality of professional opportunity by granting individual 
support rather than compensating for social inequalities by means of 
vaiying arrangements: 

- avoided discrimination. at all costs: 

- avoided preferential treatment for gifted students; 
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- activated the full potential of social demand notwithstanding the 
demands of the labour market. 

BA/dG-support was introduced with the stipulation that State support be 
subordinate to the responsibility of parents to support their children 
throughout their entire education. The principle of family contribution is 
stated in civil law and is still in force as weU as the BA/oG-principle of 
subsidiary support. 

In conclusion, it can be said that, until the beginning of the 1990s, the 
use of indicators was mainly determined by the aims of equality both with 
regard to the supply of resources and with regard to social demand. 

The elaboration and use of indicators in a system of equality had and 
still has two main functions as of the beginning of the 1990s: 

- observation; 

- planning. 

2.1. A Retrospective Observation 

The most important indicators for the observation of past developments in 
the higher education system at system-level are the key figures for higher 
education to be found in the Hochschulstatistische Kennzahlen published by 
the Federal Agency for Statistics {Statistisches Bundesamt, 2000, 2001). 
They have a long tradition and were produced in the 1970s alongside the 
process of providing open access and the evolving massifrcation of higher 
education. 

The primary reason for elaborating these indicators was the desire to 
condense the abundance of data to a small set of values briefly describing 
the characteristic stmcture of the system. The tables of absolute numbers 
of students, staff, space, and, later on, finances were aggregated and then 
subdivided to structural ratios or index ratios. 

Example: Pass rates {Pnifungseifolgsquote) “Number of examinations 

divided by four main types of examinations: universify, polytechnic, 

doctoral, and teacher examinations”. 

These stmctural ratios had more or less a primarily descriptive function. 
Their informational value increased when they were compared in time 
series. 

Still more significance was derived from so-called relative indicators, 
which related different objects: 

- the ratio of the number of students entitled to enter the higher 
education system to that of the age cohort in the population 
[Studienberechtig tenquote ) ; 

- the ratio of the number of first year students to that of the total age 
cohort {Studienanfdngerquote); 

- the ratio of the number of eligible to that of enrolled students to first 
year students {Ubergangsquote); 
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- the student/ staff ratio [Betreuimgsrelationen). 

Another type of Kennzahl or indicator was introduced '^th the intention 
of using it as a “warning light”. It was intended to deal with one of the main 
dysfunctions of the German higher education system: the growing study 
duration and the “over-aging” of graduates. The indicators in question were 
the following: 

- student mode and quartiles of study-duration [Fachstudiendaiier]; 

- arithmetic mean age of first-year students [Durchschnitts alter 
Studienarifdnger ) ; 

- arithmetic mean age of graduates (Durchschnittsalter Absolventen). 

In principle, this whole set of indicators was subordinated to the task of 
guaranteeing that access to higher education would respond to social 
demand for it, even to the extent of favouring excessive open access as well 
as academic freedom. The indicators recorded the impact and configuration 
of the resulting demand and the frictions of a process that was determined 
by free choice and the absence of steering. 

Consequently, students were viewed as autonomous input-elements, as 
externalities of the higher education system. Even their knowledge and 
personal abilities, as decisive inputs into the process of teaching, were 
neglected as independent variables. This type of competence-indicator was 
only brought to the attention of the German authorities by the OECD- 
TIMMS Study. 

The provision of study places in response to social demand were the 
main rationale behind most of these indicators. Although the indicators 
were good instruments of observation at the transition points of the process, 
nobody would have thought, at that time, of using them for the 
measurement of, for example, rates of return, effectiveness, or value added. 

Cost indicators 

It is true that as early as 1975, in Germany, a set of indicators was 
developed with a view to measuring the efficiency of higher education 
institutions. This project, launched by the Rector’s Conference within the 
OECD-IHME Programme (Westdeutsche Rektorenkonferenz, 1980), 
designed and calculated, for instance, cost-indicators for the comparison of 
study programmes of different institutions. Obviously, this competitive and 
comparative approach was alien to the strategic thinking of the time. 
Because the resulting pilot study failed to have any influence at system or 
institutional level on policy actions taken in the 1980s, it was not followed 
up until the beginning of the 1990s. 

Social indicators 

Instead, policy-makers in higher education based their information 
preferably on descriptive indicators that conveyed a general impression of 
their policy outcomes. Therefore, apart from supply-side oriented indicators. 
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social indicators were the most important tools, in the phase characterized 
by open access, by which social mobilization could be observed and proven. . 

The serious interest in social mobilization and social justice with regard 
to open access was highlighted by a full set of indicators reflecting the 
demand-side of the mobilization process. 

T^me-series of overall participation rates - as cited above as being among 
the “key figures” - were produced observing the participation rate at: 

- upper secondary school level; 

- at the transition point before entering the higher education system; 

- at first year level in higher education; 

- at the overall higher education system-level. 

Also, the participation rates of students from different socio-economic 
backgrounds were calculated. The indicators were condensed from special 
social surveys. These surveys extended back to 1951 and are still carried 
out today, every three years (BMBF, 1999). 

Social group specific educational participation rates before and after 
access to higher education were produced with regard to: 

- parental education (academics versus others); 

- family income (quartiles); 

- occupational status (blue collar worker, white collar worker, civil 
servant, and self-employed); 

- social background (combined indicator of education, income, and 
status). 

The indicator, “working class children participation rate”, was the “water 
level” indicator of social justice in the higher education system and the “acid 
test for all party-programmes for educational policy. 

As portrayed in Figure 1, below, the participation rate of working class 
children rose only slightly in the 1980s, without diminishing the scissors- 
development between upper and lower classes. 

Another key indicator in this field was the “reproduction rate of 
academics”, an indicator also mentioned in the UNESCO Framework, but as 
a performance indicator, in order to gain an indication of the role of higher 
education as a catalyst for the entire education system”. In the human 
capital growth theory, favoured by OECD, this indicator is intended to 
explain the probability for individuals whose parents have also completed 
tertiary education”. 

With regard to the use of indicators, one should note that, in Germany, 
this more or less similar indicator, “ratio of students with an academic 
fomay background to age cohort of the same background", is interpreted in 
the opposite sense than in the case of UNESCO and OECD. 

In Germany, the “catalyst” function is deemed to be the main barrier to 
the access of non-traditlonal youth to tertiaiy education. The assumption is 
that social selection is consolidated by the reproduction of academics as the 
main owners of cultural capital. 
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The “cultural capital trap view” (Boudon, 1974) becomes plausible - and 
might also be so for UNESCO - when this indicator is confronted with the 
complementary indicator of the non-academic reproduction rate, revealing a 
probability that is five times lower than the academic reproduction rate with 
a distance that has been expanding over three decades. 

The selective function of education is multiplied as it works at several 
thresholds, as the schematic presentation of social selection, the education 
funnel, shows in Figure 2. 

Figure 1: The participation of 18-21 year-olds in higher education by paternal 
occupational status over the 1985-1996 period (4*J^ threshold) 
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Figure 2, The education funnel: Schematic presentation of social selection - The 
educational participation of children of upper and lower class social background 
groups (in percentages) 
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This example of different indications offered by the same indicator raises 
the issue of content validity. It proves to be a pressing problem even with 
descriptively “neutral” indicators for observation, for it is not by definition, 
but by use, that strategic function is determined. 

2.2. Planning Indicators 

In the expansion phase of the higher education system in Germany, 
planning standards were the most important indicators for policy action. 
They were the decisive instruments in the supply of adequate resources and 
in the control of exhausted resources. 

Overall space standards regulated the size of building capacity according 
to different types of students and main fields of disciplines: 

Example: 

- 17 m2 per full-time equivalent-student in engineering disciplines, at 
universities, and 

- 5.7 m2 per full-time equivalent-student in the social sciences, at 
universities. 

“Refined” space standards described the size of different space categories 
(teaching, laboratories, communication, etc.) by which the overall space 
standards were to be broken down. 

Student/ staff ratios regulated the “density of instruction” (Betreaungsdichte} 
at all higher education institutions. 

The most decisive guideline indicator with regard to capacity building in 
the German higher education system was and still is the curriculum norm 
{Curricularnormwert-CNW}. The set of curriculum norms is applied as a 
formula: 



s = CNW * X, 



which is used for the allocation of study places. The ’’curriculum norm” 
(CNW) is the size of the teaching load requested per student in the 
respective study programme (Uthoff, 1995). 

Example: In the Classical Music study programme, CNW = 1.1317. This 
means that for every student in this programme a teaching load of 1.1317 
! hours is requested. 

The CNW multiplied by the numbers of students equals the overall 
teaching load of a department necessary to serve the demand for the 
student intake. According to the State regulation on “the exhaustive use of 
capacities”, all departments offering programmes with numenis clausus 
have to accept the full number of applicants according to their teaching 
capacity. 
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The indicator formula behind the CNW-norm, which assumes the 
intention to exhaust the potential of capacities, also implies the 
fundamental aim that capacities should be provided and utilized 
consistently and evenly all over the country. 

The uniformity of study place provision and its utilization has to be 
viewed as a unique feature and characteristic of the higher education 
system in Germany. It guarantees the broad distribution of high quality and 
the equal supply of social demand. 

Taking the importance of all “planning indicators” into account, one has 
to ask if these standards and norms, which are disguised as indicators, 
fulfill the definition of what an indicator is. Guideline indicators do not 
“convey a general impression of the state of the situation”, but structure the 
future state of the situation (Kaiser, 2003, in this volume, pp. 31-35). 

In a higher education system which aims at the equal and even provision 
of study opportunities, indicators have a strong strategic impact on action. 
For this particular reason, norms and standards should be considered 
when “strategic indicators” at system-level are wanted. Frans Kaiser, too, 
draws attention to this function, when he states that strategic indicators 
should be “prospective”. 

This author is not quite certain if Kaiser had in mind these normative 
indicators when mentioning the “prospective” quality as additional criteria 
for strategic indicators. 

Another reason for insisting that Richtiverte or planning norms be 
treated as a significant type of indicator is the fact that norms of standard 
provision or standard utilization can also be used to confront potential 
provision and utilization with real performance. The ratio of real to potential 
intake or output provides the capacity utilization rate of the system as a 
scale for assessing productivity. 

Being governed by the paradigm of the equal opportunity policy of the 
1980s, institutions of higher education used the capacity utilization rate 
primarily for purposes of adjustment. For instance, when they operated over 
capacity, their requests for more resources were based on this indicator. 

The capacity utilization ratio proved to be the key indicator for the 
incremental growth of the demand-oriented policy of open access. When 
that policy changed, the purpose of this indicator also changed. The 
capacity utilization ratio is no longer a growth planning tool. It now serves 
as a scale for assessing productivity, the focus of the next section. 

3. NEW MANAGEMENT INDICATORS 

The new paradigm of the market and new management issues in higher 
education immediately brought about the necessity that indicators focus 
more on these core goals of the higher education system. They changed in 
purpose and perspective. 

The purpose of the use of indicators is no longer that of controlling the 
evenness of the distribution of resources but that of revealing differences, 
ranks, and variability, and the reasons for differentiation. 
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Differentiation is taken as the expression of competition and competition 
as the instrument to assure quality and productivity without defining - at 
the risk of designating losers and winners - the optimum state of the quality 
of higher education in Germany. In this new perspective, the following 
issues - achievement, deregulation, globalization, and rationalization of 
costs - are regarded as crucial points of reference for the definition and use 
of indicators at system-level. 

3. 1 . Achievements 

The benefits of higher education are the crucial point of assessment. In the 
absence of data on financial benefits (profit), a set of output indicators is 
configured: 

- the ratio of graduates to first year students; 

- the ratio of graduates to staff members: 

- the ratio of successful completion to drop-out; 

- the proportion of graduates in the labour-force; 

- the rates of employment and unemployment of graduates: 

- the average amount of time a graduate needs during which to find 
employment; 

- the current rate of expenditure per graduate: 

- the current rate of expenditure per graduate in terms of the planned 
completion time. 

The ratio of graduates to input dimensions (staff, time, and costs) serves 
as a substitute for measuring performance effectiveness. The labour force 
indicators serve as substitutes to measure the benefits of human capital 
investment. 

Currently, in Germany, there is little interest in and a lack of 
methodology for using indicators for quality assessment. Peer review is 
hardly ever broken down into ordinal indicators (ranking lists). In the field 
of teaching at the level of higher education, the value added is neither 
assessed nor quantified in relative indicators. 

3.2. Deregulation 

Most of the indicators mentioned above are generated at institutional level. 
At system level, they are either used at highly aggregated level with regard 
to national framework planning and the setting of overall goals, or - and 
more frequently - for the bargaining process with institutions: for example, 
distribution of lump sum budgets to institutions by performance-funding or 
contract-funding. This necessity explains the preoccupation with cost- 
indicators. 

These indicators serve as: 

- a measure for the performance-oriented differentiation of funding for 
institutions and programmes; 

- criteria for goal- achievement in contract funding; 
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- a means to designate criteria in incentive funding for selected 
innovative institutions. 

To give an example, the source sheet for the Chemistry study programme 
is derived from a systematic collection of indicator sheets for every study 
programme offered by universities in five German Lander fTable 1). 

At system-level, these indicators - primarily the costs per graduate 
indicator - are taken into account in deriving standard price clusters for 
State performance funding for institutions, without directly interfering with 
institutional management. Of course, considerations go further and will also 
affect the single institution: cutting costs, regional concentration of 
programmes, and merging of institutions. 



Table 1. Programme-sheet for Chemistiy; performance indicators 



Chemistry 


Costs (in DM) 


Stciff (full-time 
equivalent) 


Costs for instruction (in DM) 


Programme 


Overall 


Direct 


Indirect Profs. Others Total 


Total 


Study- 

place 


Per Per 

student graduate 


University of 
Hambui^ 
University of 
Bremen 


28,700,353 


25,027,511 


3,672,842 28.2 


70.4 


98.7 


13,034,230 


13,827 


19,759 


85,983 


12,932,671 


6,342,131 


6,590,541 17.0 


28.0 


45.0 


5,345,031 


9,718 


27,194 


179,206 


University of 
Kiel 


13,479,915 


11,416,916 


2,062,999 12.0 


39.9 


51.9 


5,244,041 


9,053 


18,960 


58,083 


University of 
Oldenburg 
University of 
Gottingen 
Technological 


18,772,547 


8,306,592 


9,522,133 13.0 


27.0 


40.0 


6,904,880 


14,998 


22,404 


117,416 


28,960,252 


21,531,520 


6,535,615 22.0 


67.0 


89.0 


10,954,066 


16,503 


16,383 


71,988 


University of 
Braunschweig 


16,507,187 


12,429,052 


2,767,650 18.0 


63.8 


81.8 


6,675,613 


9,176 


18,170 


52,807 


University of 
Hannover 


19,561,529 


12,671,318 


5,458,404 22.0 


54.0 


76.0 


7,580,573 


9,811 


15,107 


59,130 


University of 
Rostock 


10,379,239 


8,583,771 


1,795,468 12.7 


54.0 


38.3 


4,108,976 


7,834 


38,258 


279,361 



Source: HIS, Ausstattangsvergleich norddeutscher Lander (2001). 



3.3. Globalization 

International indicators for the national comparison of different features of 
the higher education system have always been very popular in Germany, 
but more or less without any relevance for system-level decisions. This 
situation has changed since globalization became the core issue of 
internationalization. The concern for international competitiveness has 
caused the development of complex indicator- sets on international mobility 
(in- and outgoing mobility of students and staff [DAAD, 2001]) and has 
made the authorities more sensitive to the reception of indicators in the 
sphere of the national economy and of competency-development. OECD- 
EAG-indicators on “Financial and Human Resources Invested in 
Education”, especially category Bl, “Educational Expenditure Relative to 
Gross Domestic Product”, and those relating to “Student Achievement”, 
especially FI, “Mathematics Achievement of Students”, and F2, “Differences 
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in Student Attitudes Towards Science”, aroused self-critical interest and 
gave rise to further analysis and promotion-programmes (OECD, 2000). 

The traditional indicators regarding access to education and 
participation, formerly viewed in the context of the basic principle of 
freedom of education, tend to be increasingly viewed in light of human 
capital investment and the worldwide competition of knowledge -based 
economies. 

The challenging questions posed by UNESCO with regard to performance 
indicators refer only partially to the competitive side of internationalization. 
The rigours of worldwide competition and of self-interest in the use of these 
indicators are not made clear. Thus, illusions of a vision of balanced 
development among nations or between the principle of equity and the 
challenge of modernization are fostered. 

3.4. A Student-Centered Approach 

One of the fifteen areas designated in the UNESCO Declaration, Higher 
Education in the Twenty-First Century, as a priority field is the concept, “to 
place students and their needs at the center of national and institutional 
decision-makers’ concern” (UNESCO, 1998). 

This view is playing an increasingly important role in German 
discussions on higher education system development; however, it is 
predominantly couched in the terms of market considerations. In terms of 
market forces, students are called clients or consumers. In terms of 
modernization, they are regarded as allocation factors. 

Indicators on: 

- study-preferences for disciplines; 

- transition rates between different levels of education; 

- regional catchment areas; 

- migration rates among Lander, 

- success rates of mobilization schemes; 

- life-time income of graduates; 

are examples of indicators urgently needed for policy decision-making at 
system-level. Until now, there has been only a volatile set of indicators. 
Indicators are rather time-bound and problem-centered. They are neither 
connected with a theory of human capital investment, nor do they form an 
explanatory cluster with other indicators. Highly abstract OECD-EAG 
indicators, like “educational expectancy”, which refer to a model of human 
capital investment, have found little acceptance in Germany. 

4. TENTATIVE APPROACH TO THE RECONCILIATION OF OPPOSING 
STRATEGIC INDICATORS 

In the case of Germany, the referential character of indicators can be 
demonstrated very clearly by confronting their use under shifting and 
opposing paradigms. Referential implications go much further than to the 
pure, direct indication-function of indicators (present situation, core goals. 



98 



K. SCHNITZER 



and trends). The indication is strongly directed by the visions and strategies 
in force. Each vision implements its own set of indicators or even changes 
the face value of the same indicator. The indicative meaning of participation 
rates can be as different as the two sides of a coin. 

At first sight, this double-bound substance of indicators should make 
anyone using them suspicious, especially when one is aware of the fact that 
most of the ideological connotations are conveyed in a hidden agenda. 

In Germany, where two different visions of the function and development 
of the higher education system compete with each other and in reality have 
to co-exist, as both are codified by effective law and regulations, the use or 
abuse of apparently contradictory sets of indicators can be analyzed very 
clearly: yet, the outcome of this conceptual controversy is not clear. 

A good example of this dialectical relationship is the application of 
indicators pointing, on the one hand, to the obligation of the State to 
provide study places of sufficient size and of equal qualify in order to satisfy 
social demand and, on the other hand, the application of indicators which 
aim at the market-oriented regulation of the provision of study places. The 
optimal use of the capacity of the higher education system is the goal 
function of both. 

In a portfolio-presentation, these two competing aspects can be 
visualized and related to each other. To take up the example of “Chemistiy” 
once again: The source-sheet portrayed in Table 1 indicates the variation in 
teaching costs per graduate in the different chemistiy programmes at the 
ten universities concerned. 

The variation in costs per graduate among the single programmes is 
extremely large. Production costs at the University of Rostock are six tirnes 
higher than at the University of Bremen, as shown in Figure 3. If the values 
are clustered in terms of “over average” and “under average” costs per 
graduate, four universities are running their programmes with very low 
current costs per unit: another four, with high costs per unit. 

Most of the variation can be explained by regional differences in the 
demand for study places. The Chemistiy programme at the University of 
Rostock has only recently been opened. It is up to date: however, it has few 
applicants and has had very few first graduates. The universities with low 
unit costs are very attractive for applicants because of their size and the 
variety of their programmes. As long as applicants for study places can 
decide themselves where they will study, the German higher education 
system will have to live with the effects of under- and overload capacity and 
a respective variation of unit costs. Thus, the graph demonstrates on the x- 
axis the price of free choice of where to study. 

Another perspective on cost analysis is portrayed by asking to what 
extent the study places provided vaiy in terms of costs, without taking into 
consideration the utilization of these study places. On the y-axts, the 
current unit costs per installed study place are positioned according to their 
level. The variation is modest compared to utilization costs. The highest 
costs per unit are only the double of the lowest costs. This result could be 
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expected according to the principle of provision of equal study 
opportunities. 

Figure 3. The Chemistry/cost portfolio in selected universities in Germany: the 
provision of study places and their utilization 
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Source: HIS-Ausstattungsvergleich ( 2001 ). 



The unit costs are again clustered by drawing a line between 
programmes with above average unit costs and below average ones. By 
combining the distribution of unit costs for graduates and for study places, 
the strategic question of whether or not increased investment in study 
places improves utilization and by so doing reduces the production costs of 
graduates can be partly answered. With the exception of the three 
universities in the segment of low capacity costs and low production costs, 
there seems to be an exponential correlation: the higher the capacity costs, 
the lower the production costs. 

On the other hand, it might to be worthwhile analyzing the reasons for 
the existence of a “cheap cluster” (low input costs and low costs of 
production) through further investigations and additional indicators. 

It would be beyond the scope of this article to continue with an 
interpretation of the empirical values. The aim of this exercise has been to 
demonstrate the heuristic value of bringing together sets of indicators that 
represent competing strategies for the steering of a higher education system 
and that have to be reconciled in a given higher education system having 
pluralistic aims. Moreover, the effort has aimed to offer an example of: 

i) the confrontation of the two basic strategies that together constitute 
the shape of the higher education landscape in Germany: 

ii) how the effects of these two strategies can be made visible by using 
different sets of indicators in a relational perspective: 
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Hi) how, by the implementation of this portfolio approach, priorities of 
action can be deduced from this way of mapping the indicators of. 
competing goal systems. 

5. CONCLUSION 

With regard to the two competing strategies determining the higher 
education system in Gennany, one can still claim that open access remains 
the major goal of higher education policy and that, on the basis of this 
consensus, market-mechanisms, in German society, are the adjusting 
counter-weights that protect against over-bureaucratization and serve as 
the warranty of effectiveness, quality-control, and new impulses. 
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VIL Strategic Indicators for Higher Education 

Systems: Lessons from the French Experience 

THIERRY CHEVAILLIER 

Indicators are needed in order to assess the relative position of a country or 
to measure progress towards goals that have been set at a national or 
regional level in relation to missions assigned to higher education as a 
whole. They enable decision-makers to make decisions about action and to 
monitor the effects of a given policy. 

A system comprises various components. Each component fulfills a 
function or a set of functions that limits its activity and its responsibility to 
a specific domain. The actions expected from such components, the goals 
that are given, are in a way subordinated to those of the whole system. 

A higher education system is composed of operational units, universities, 
and other institutions of various kinds, with the task of delivering education 
and conducting research. It also includes agencies (ministries, funding 
bodies, research councils, and student welfare agencies) responsible for the 
co-ordination of some or all of the activities of the operational units. The 
relative weight of institutions and of agencies in a higher education system 
varies from one countiy to another. Highly centralized systems give a wide 
range of decision-making powers to coordinating agencies. Highly 
decentralized systems confer a large degree of autonomy to operational 
components. 

“System level” indicators are sensitive to the structure of the higher 
education system. Although recent years have witnessed some convergence 
in the structures of national systems of higher education, there is still a 
significant diversity that makes it difficult to devise a set of indicators that 
could be applied to all the countries of the world. 

As the various national systems face different constraints and unequal 
levels of development, it is likely that the priorities of countries will vary. 
What is strategic in the most advanced countries can be of little interest for 
the less advanced ones. Countries with more than half an age class enrolled 
in higher education obviously look for indicators that are far more 
sophisticated than those sought by countries that are still struggling to 
achieve universal basic education. 

For a long time, basic indicators have been available for nearly eveiy 
countiy in the world. These include student enrollments, graduation rates, 
and the resources allocated to higher education, like public funding and 
staff, etc. As the world enters the Twenty-First Century, it seems that this 
information is no longer adequate. The rapid development of the information 
and communication technologies in recent decades has altered both the 
needs of society and the means it can use to satisfy them. At the same time, 
the patterns of organization of society have also evolved. Higher education 
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has been given new missions and must confront new situations. New actors 
have entered the field and consider that they have a stake in it. Institutions . 
are faced with competition from new providers and have to rethink the way 
they operate. Regulators and coordinating agencies are having to deal with 
increasingly complex systems with constantly shifting boundaries. New 
indicators are needed. 

1. THE EXTENSION OF THE MISSIONS OF HIGHER EDUCATION 

The traditional missions of higher education were basically the education of 
elites, the advancement of knowledge, and the preseiwation of culture. 

Owing to the ever-increasing division of labour and specialization of 
knowledge, industry turned to higher education institutions and required 
that they contribute to economic development by raising skill levels and 
ensuring technological progress in all sectors of production. In particular, in 
recent years, higher education has been expected to support an expanding 
knowledge-based economy and to prepare for geographical and professional 
mobility for a smoother operation of labour markets. 

At the same time, as social and human capital came to be a key factor in 
the creation of wealth and the promotion of welfare, higher education came 
to be viewed as a means for bringing about equality of opportunity among 
individuals and social groups. 

2. MULTIPLE STAKEHOLDERS 

After centuries of relative isolation, higher education has been attracting 
wide attention from the public. As the size of systems increase, new 
stakeholders appear and tiy to influence policy to put foi'ward their specific 
goals. Academics, students, public agencies, industry, and local 
governments have been playing increasingly important roles in decision- 
making processes or trying to interfere with them. In many European 
countries, the funding of higher education has been diversified, enabling 
new stakeholders to influence the steering of institutions or of whole 
systems, provided they contribute resources. They too are interested in 
monitoring developments that affect them and are demanding new 
indicators. 

3. TRENDS AND POLICIES IN EUROPEAN, PARTICULARLY FRENCH, 
HIGHER EDUCATION SYSTEMS 

The last quarter of the Twentieth Century witnessed dramatic developments 
in higher education in most of Western Europe. Some of these developments 
were the direct consequence of the building of the European Union. 

There has been the fact of democratization. In several countries, the 
proportion of a generation gaining access to some form of higher education 
jumped from less than 10 percent to about 50 percent. As a result, the 
proportion of graduates in the adult population has been growing steadily. 
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The move from elite to mass higher education has raised issues such as 
that of the lowering of the quality of education and of the downgrading of 
the employment possibilities available to graduates. 

Such a large expansion in student numbers was matched by the creation 
of appropriate jobs only because higher education had been diversifying and 
differentiating itself. New programmes have been opened on the basis of the 
current or foreseen needs of the economy for new qualifications and 
competencies. Similarly, new types of institutions have been created when 
traditional universities were reluctant, or badly equipped, to embark on the 
type of diversification that the new institutions represented. Such action, in 
a way, has prevented a downgrading of the jobs offered to the rapidly 
growing number of graduates. However, it has created difficulties in regard 
to the assessment and maintenance of quality with the customary tools and 
concepts of traditional universities such as academic standards and peer- 
reviews. 

The diversification and differentiation of higher education occurred 
mainly in sectors in which education and training for specific employment 
could be made more relevant by bringing in professionals who could 
participate in teaching and advise academics on curriculum. The expansion 
of such programmes is referred to in France as the professioncdization of 
higher education (perhaps vocationcdization would more aptly describe this 
phenomenon). Most of the innovations in curricula that have been 
introduced into French universities in the past thirty years have been of 
that kind. 

Professional programmes, in contrast to the traditional course 
programmes for which little has been changed in the contents and methods 
of teaching, have attracted a growing number of students. These 
programmes cover a wide range of different types: short programmes (2 
years) in the University Institutes of Technology (lUT) created in the late 
1960s, undergraduate programmes like the Master’s Degree programmes in 
Science and Technology (MST), and the offerings of the University 
Professional Institutes (lUP), or one-year postgraduate programmes such as 
the Higher Diploma of Specialized Studies (DESS). Their common feature is 
the participation of industry in the conception of curricula and in the actual 
teaching and training, either by providing part time and associate lecturers 
or by taking students into industrial internships (stogfes). 

They also contrast with other programmes as far as access is concerned. 
While entrance to most French university course programmes is subjected 
only to the possession of a baccalaureat, the end-of-secondaiy-education 
diploma, professional programmes select their students at various stages of 
their studies. 

The development of professional programmes has had a far-reaching 
effect on higher education. At various stages of their academic careers, 
students are offered “bridges” to enter different programmes, either in their 
universities or in other higher education institutions. The multiplicity of 
student “trajectories” makes it difficult to assess the efficiency of higher 
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education at institutional level, for students transferring from one 
institution to another may be treated as drop-outs. 

The expansion of continuing education and training has also had a 
strong impact on the French higher education system. Many professional 
courses were first introduced experimentally as continuing education 
programmes. The principles of lifelong learning were recently translated into 
policy when universities and other higher education institutions were 
advised to review their programmes in order to facilitate the enrollment of 
continuing education students alongside “traditional” students. The task 
implied the introduction of modularity and flexibility in the way courses 
were organized. This channel is probably one of the main ones through 
which the use of the new information and communication technologies will 
be introduced into most institutions. The pressure on universities to open 
themselves to continuing education was stimulated by legislation providing 
for the accreditation of professional experience towards access or even the 
award of degrees. 

The reseai'ch sector of French higher education has also been affected by 
deep transformations. Research funding was separated from funding for 
teaching and made more dependent on competitive bidding on the basis of 
projects. The diversification of resources was encouraged with the effect of 
establishing new working relationships between industry and the academic 
world. Contract research money has come to play a significant role in the 
realization by academics that they are actors in the economic development 
of the country. University industrial subsidiaries, science parks, and small 
business incubators have become strong elements of the national policy to 
promote industrial innovation. 

The internationalization of higher education took a new start with the 
creation of the ERASMUS student exchange programmes. It has slowly 
percolated through institutions and programmes. The new foreign student 
population had different needs and different purposes. As they were on 
short stays (one semester to one year), they insisted that their studies in a 
foreign institution be recognized in their own countries. They brought 
pressure to bear on faculty and academic authorities to introduce 
transferable credits and to engage in negotiations with their foreign 
counterparts to assess academic work on a comparable basis. This process 
strongly contributed to harmonization and convergence, both inside 
national higher education systems, in which many disciplinary 
idiosyncrasies could be found, and among countries. With more than 
30,000 students sent abroad in 1998 (and even more received from abroad), 
European academic exchanges have had an impact on every institution and 
every programme. 
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4. IMPLICATIONS FOR THE CONCEPTION AND USE OF INDICATORS AT 
SYSTEM LEVEL 

To a great extent, the phenomena occurring in higher education systems 
have become increasingly complex. Growing complexity challenges existing 
indicators. The use of simple indicators to compare the state of a system at 
different points in time may hide significant developments. Similarly, in 
international comparisons of higher education systems, the usual indicators 
may produce statements that appear, to an informed observer, to be 
obviously wrong. One way to resolve this dilemma would be to try adding 
new indicators more suited to specific situations. However, the information 
that could be gained from increased precision might turn out to prevent 
reliable comparisons. 

In many countries, indicators have been devised which accurately 
describe idiosyncratic practices or situations. When put into international 
perspective, they no longer make sense. It is only when systems converge 
that it is possible to increase the amount of significant information by 
introducing new and more detailed indicators. Such a convergence is slowly 
taking place throughout the world in some of the activities performed m 
higher education systems. It can be accelerated when policy harmonization 
is decided upon inside a regional group of countries. At the European level, 
the common decision to create a European Higher Education Area will 
certainly have an impact on the definition and use of higher education 
indicators. 

A rapid review of indicators that have been used or proposed in the 
French context for education and training (and leaving aside research and 
the other functions of higher education) will help to convey the point. 

4. 1 . Financial Indicators 

The use of financial indicators to assess a relative position or a degree of 
evolution is founded on the assumption that the value of the resources 
devoted to higher education reveals the quantity of higher education 
produced by a given country at a given time. A larger expenditure is 
presumed to be equivalent to more education, a smaller expenditure, to 
less. This assumption is acceptable if three broad conditions are met: (i) The 
level of economic development and the cost of the various inputs in the 
countries compared must be roughly similar, (ii) The use of resources must 
be measured in the same manner. (Hi) The type and the quality of education 
must be comparable. 

Comprehensive accounting for the resources used is almost impossible. 
Apart from public expenditure as measured by budgetary appropriations - 
which are quite well known in nearly all countries - it is usually difficult to 
rely on information available from other sources. In a few countries, a large 
portion of the total expenditure on education is borne by institutions that 
collect funds from various origins. Institutional accounts provide good 
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coverage of expenditures. In other countries, many similar costs are borne 
by different agents that provide funds or services to students or to. 
institutions. A comparative analysis of student aid in the countries of the 
European Union conducted by EURYDICE has identified problems arising 
from the juxtaposition of various schemes funded by different 
administrations and local governments with possible cross-subsidization. 

At a time when funding is becoming increasingly diversified, such 
problems are likely to increase. They are usually solved by opting for a 
narrow concept of educational expenditure that is easy to measure [but may 
be judged by experts as misleading). It is likely that the wider the group of 
countries to be compared, the narrower the concept. 

An attempt to overcome the heterogeneity of systems in international 
comparisons is to calculate an indicator of effort by relating the amount 
spent to the total available resources; a ratio of higher education 
expenditure (public or total) to the total of public expenditure or the GDP. 
Such an indicator, which informs about the priority given to higher 
education by the country or its government, presents the same 
methodological flaws as does total expenditure. 

When the size of the student population varies, a usual correction 
consists in calculating public or total expenditure per student. The view is 
that a student is a student regardless of the type of course programme in 
which he or she is enrolled. The introduction of short courses or the 
expansion of postgraduate programmes which directly affects the number of 
students enrolled in a national system will suggest an improvement or a 
worsening of the indicator. 

Looking at higher education as an investment suggests the need to relate 
the total cost per student until graduation (the cost of a graduate) to income 
per capita. Production of a higher education graduate could represent twice 
or twenty times the annual average income of the population. It is 
undoubtedly a better measure of effort, but it does not indicate who bears 
the weight. If one knows the distribution of funding between public and 
private sources, one obtains a better idea of this effort. 

4.2. Input Indicators 

Indicators that relate to the various inputs of higher education, staff, 
equipment, and infrastructure are similar to financial indicators in that 
they usually infer quality or quantity of education from the amount or value 
of what is used to provide education. They can be reliably applied to 
comparisons in space or in time, if the structure of the inputs is stable and 
uniform. Staff-student ratios and numbers of computers per staff member 
or per student convey information when the related inputs are used in the 
same way. At a time when one type of input is substituted for another, 
information on volume or quality of the educational output cannot be 
inferred from such indicators. 

As the new infonnation and communication technologies make their way 
into higher education, such substitution is likely to take place. The 



LEARNING FROM THE FRENCH EXPERIENCE 1 09 

observation of individual institutions has shown that, for a time, the new 
ITCs are only added to existing inputs, for old teaching practices are 
retained alongside the new ones. Substitution starts only after a period of 
time. It is extremely difficult to get an idea of the rhythm at which this 
process takes place, especially at system level. 

4.3. Indicators of Efficiency 

Efficiency refers to the extent to which an organization achieves its goals. 
Economists customarily distinguish internal efficiency from external 
efficiency according to the level of the goals that are considered. Applied to 
the operation of a higher education system, internal efficiency might 
address a question such as: Does the system produce the highest number 
of graduates given the available resources? External efficiency, however, 
might ask if the system produces benefits to society that are sufficiently 
large to justify the value of the resources expended. 

The indicators of the internal efficiency of the educational process 
include such data as duration of studies, dropout rate, graduation rate, etc. 

Indicators of external efficiency pose questions about the relevance of 
education by looking at imbalances on the labour market and mismatches 
between qualifications produced and needed resulting in the unemplo 3 rment 
of graduates, shortages of qualified labour, or the downgrading of graduate 
emplo 3 rment. 

The first t 3 rpe of indicators of efficiency is made increasingly difficult as 
the individualization of student trajectories develops. When there is only one 
model of programme for the whole of higher education, say the BA or the 
BSc in three years, it is quite simple to measure dropout rates and to 
interpret them as signs of inefficiency. When students are allowed to 
reorient themselves and to embark on further studies at different stages of 
their student careers, measurements of failure or attrition are tricky, and 
conclusions about efficiency are dubious. The tendency is to stick to 
“traditional” trajectories even when they represent only a small part of the 
actual situation experienced by students. Lifelong learning, implying 
students moving in and out of individual institutions or of the system 
altogether, will certainly not clarify the picture. 

Indicators of external efficiency are equally flawed. When looking at the 
labour market, it is difficult to disentangle what the result of the education 
system is from what the result of the economic situation of a country or of 
an individual industry is. In a few cases, a market situation can be 
considered as the product of an inefficient system, e.g., graduate 
unemplo3rment that endures over a period of economic expansion. But in 
most cases, no such clear answer can be given. The same would apply to 
variations in the length of the period between graduation and first 
employment. 
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4.4. Quality Indicators 

In a way, external efficiency amounts to quality as judged by the labour' 
market. Higher education has other purposes that could lead to a different 
and wider meaning for quality. The trouble is that these purposes are 
difficult or impossible to quantify. Student satisfaction could be an 
approach to quality, but, if applied to the period of study, the result would 
bear little relation to the role that society assigns to education. If viewed 
through an assessment of job satisfaction and the appropriateness of 
knowledge and competencies acquired to work and live well, such an 
indicator would make more sense, but the verdict would still be likely to be 
influenced by wider or unrelated circumstances. 

Feeling oneself able to adapt to new situations and being equipped to 
move from one country to another certainly denote a quality education but 
of a type, unfortunately, quite difficult to translate into synthetic indicators 
that would be comparable through time and space. 

Nevertheless, there is a need for qualitative indicators to measure the 
value that society sets for higher education missions and goals that go 
beyond the needs of the labour market and how well higher education 
institutions achieve these goals. 

4.5. Indicators of Access and Equity 

In the Western world, where mass higher education has been achieved, 
access and equity have become a less pressing concern at system level. In 
France, women enter higher education and graduate in larger numbers 
than do men. The same is true of students from different social 
backgrounds. Differences and possible discrimination occur within specific 
institutions or sectors of higher education. As the hierarchy of prestige for 
institutions or programmes varies widely from country to country, there is 
no point in elaborating indicators that would enable comparisons of 
different countries. It is, however, interesting and important to develop 
appropriate indicators in each country in order to monitor the respective 
internal situation. 

5. CONCLUSION 

Contrary to the impression conveyed by the numerous limitations and 
problems to which this article has pointed with regard to the elaboration of 
indicators for use at system level, such indicators are useful and should be 
developed. However, it is futile to try to design a battery of sophisticated 
indicators that could be applied to all systems of higher education. The 
already existing indicators, if more accurately calculated, could satisfy the 
needs of decision-makers in a number of countries. Highly specific 
indicators will still be needed by individual countries to keep track of 
policies adapted to their peculiar contexts. Within groups of countries with 
a sufficiently high degree of homogeneity, it is essential to produce new 
common indicators. 
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1. INTRODUCTION 

From an engineering point of view, the Polish system of higher education 
may be considered as a complex, hierarchical control system. According to 
the engineering art, the design of such a system and its effective operation 
requires a precise definition of the goals of its functioning and reliable 
information as to its status, as well as mathematical measurements of the 
rate of achievement, te., of the abstract distance between the current state 
and the desired state of the system. System-level indicators for 
tertiary/ higher education are thus indispensable for the efficient 
management of the national system of higher education. 

The diversity of the systems of higher education in various countries 
makes it practically impossible to understand the statistical indicators 
corresponding to a given national system outside of the system in question, 
without some explanatory information on the specific system and 
indicators. This principle is applicable, in particular, to the Indicators 
currently used by the Central Statistical Office [Gldwny Urzq.d Statystyczny 
- GUS) in Poland. They are published every year by the end of September in 
a separate volume tided Higher Schools and Their Finances in... (here the 
year follows), and contain data as of 31 December, of the immediately 
preceding year e.g., Central Statistical Office (2000). 

Two groups of indicators characterize the Polish institutions of higher 
education, viz.: non-financial and financial indicators. Both groups are 
composed of simple indicators “which are expressed in the form of absolute 
figures and are intended to provide a relatively unbiased description of a 
situation or process” (Abercromby and Fielden, 2000). Below, certain basic 
categories are characterized that are used for straicturing the statistical data 
in Higher Schools. ... 

There are fourteen types of institutions of higher education in Poland, 
referred to hereinafter, for simplicity, as higher schools. These are, inter alia, 
universities, technical universities, business schools, medical academies, 
and maritime academies. They are classified according to their locations, 
Le., the town and the province (called a wqjewddztwo in Polish), as well as 
according to their owners: State schools and non-State schools. 

Polish higher schools offer four types of studies, viz.: regular studies (also 
called full-time or day studies), evening studies, extramural studies, and 
external studies. They may lead to BSc, MSc, or PhD degrees. The 
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programmes leading to BSc or MSc degrees are offered in about one 
hundred study fields, classified into twenty-one groups of study fields • 
(according to the International Standard Classification of Education). The 
programmes leading to PhD degrees are offered in about sixty-two scientific 
disciplines, classified into seventeen branches of science; e.g., chemical 
disciplines, technical disciplines, and medical disciplines. 

In the total population of students, the following groups are 
distinguished for statistical purposes: foreign citizens, foreign citizens of 
Polish origin (the so-called Polonia), women, handicapped persons, and age 
groups (18 and less-than-l8-year-olds, 19-to-29-year-olds, 30 and over-30- 
year olds). 

The employees of higher schools are subdivided into academic staff 
members (professors, associate professors, lecturers, etc.), and non- 
academic staff members (engineers, technicians, administrators, 
secretaries, etc.). They are also differentiated according to the formal 
qualifications they hold that have been confirmed by scientific titles and 
degrees. Two scientific degrees are awarded in Poland, Le., the PhD, being a 
regular doctoral degree, and the DSc, being a senior doctorate (the so-called 
habilitation). The BSc and MSc degrees are considered to be professional 
degrees, not scientific degrees. These degrees are awarded in scientific 
disciplines by the scientific councils of the faculties of higher schools or of 
research institutions of non-academic type. The highest academic 
qualifications are formally confirmed in Poland by the title of Professor, 
which is awarded by the President of the State. 



2. NON-FINANCIAL INDICATORS 



Part I of the document Higher Schools... lists fifty- three non-financial 
indicators (the table numbers indicated in the parentheses refer to Part I of 
the publication by the Central Statistical Office, (GUS, 2000), viz.: 



AIDl 

AID2 

CONI 

EMPl 

GRAl 

GRA2 

GRA3 

GRA4 



students receiving financial aid by kind of aid, type of school, and 
wojewodztwo (table 58); 

students in hostels and using the services of student food 
services by type of school and wojewodztwo (Table 59); 
post-graduate non-degree courses (Table 19); 

academic staff and non-academic staff by type of school, position, 
and type of employment (Table 31): 

graduates of MSc-level and BSc-level studies by type of school 
(Table 6): 

graduates by type of school (Table 7): 

graduates of MSc-level and BSc-level studies by wojewodztwo 
and type of school (Table 8): 

graduates of MSc-level and BSc-level studies by groups of study 
fields (Table 9): 
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GRAS - 

GRA6 - 

GRA7 - 

GRAS - 

GRA9 - 
GRAIO- 

GRFl - 

GRF2 - 

GRF3 - 

GRF4 - 

sen - 

SCI2 - 

SCI3 - 

SCI4 - 

SCI5 - 

SCI6 - 

SCI7 - 

; SCI8 - 

SCI9 - 
STFl - 

STF2 - 

STF3 - 



graduates of regular studies by type of school, group of study 
fields, and study field (Table 45): 

graduates of evening studies by type of school, group of study 
fields, and study field (Table 46); 

graduates of extramural studies by type of school, group of study 
fields, and study field (Table 47); 

graduates of external studies by type of school, group of study 
fields, and study field (Table 48): 

graduates by group of fields of study and study field (Table 49); 
graduates of non-State schools by group of study fields and study 
field (Table 50); 

foreign graduates by type of school, group of study fields, and 
study field (Table 51): 

foreign students enrolled in regular studies, by age and type of 
school (Table 52); 

foreign students pursuing evening studies, by age and type of 
school (Table 53); 

foreign students enrolled in extramural studies by age and type of 
school (Table 54); 

PhD students by type of school (Table 20); 

PhD and DSc scholarships for persons preparing their theses 
(Table 21); 

PhD and DSc degrees (awarded by all institutions) by 
wojewodztwo and type of institution (Table 22); 

PhD and DSc degrees (awarded by all institutions) by branch of 
science (Table 23); 

PhD and DSc degrees (awarded by all institutions) by type of 
institution and branch of science (Table 24); 

PhD degrees (awarded by all institutions) by age of recipient and 
branch of science (Table 25); 

DSc degrees (awarded by all institutions) by age of recipient and 
branch of science (Table 27); 

professor’s titles (awarded by all institutions) by uxDjewodztwo 

and type of institution (Table 29); 

professors’ titles by branch of science (Table 30); 

foreign students and graduates by type of school and group of 

study fields (Table 13); 

foreign students and graduates by wojewodztwo and type of 
school (Table 14); 

foreign students and graduates of Polish origin by type of school 
and group of study fields (Table 15): 



114 



R. Z. MORAWSKl 



STF4 - 

STF5 - 

STF6 - 

STHl - 

STH2 - 

STH3 - 

STUl - 
STU2 - 
STU3 - 

STU4 - 
STU5 - 
STU6 - 
STU7 - 

STU7 - 

STU8 - 

STU9 - 

STUIO - 

STUll - 

STU12 - 

STU13 - 

STU14 - 

STU15- 
STU16 - 



foreign students and graduates of Polish origin by wojewodztwo 
and type of school (Table 16); 

foreign students by type of school, group of study fields, and 
study field (Table 17); 

foreign students and graduates by continent and countiy of 
origin, including foreign students of Polish origin (Table 18); 

handicapped students pursuing regular studies by type of school, 
group of study fields, and study field (Table 55); 

handicapped students enrolled in evening studies by type of 
school, group of study fields, and study field (Table 56); 

handicapped students pursuing extramural studies by type of 
school, group of study fields, and study field (Table 57); 
students by type of school (Table 1); 
freshmen by type of school (Table 2); 

students in higher schools by wojewodztwo and type of school 
(Table 3); 

freshmen by wojewodztwo and type of school (Table 4); 
students by group of study fields (Table 5); 
newly-admitted students by type of school (Table 10); 

students in extramural studies by age and type of school (Table 
44); 

students in regular studies by type of school and group of study 
fields (Table 32); 

students in regular studies by town, type of school, and 
wojewodztwo (Table 33); 

students in evening studies by type of school, group of study 
fields, and study field (Table 34); 

students in evening studies by type of school and wojewodztwo 
(Table 35); 

students in extramural studies by type of school, group of study 
fields, and study field (Table 36); 

students in extramural studies by type of school and 
wojewodztwo (Table 37); 

students in external studies by type of school, group of study 
fields, and study field (Table 38); 

students in external studies by type of school and wojewodztwo 
(Table 39); 

students in regular studies by age and type of school (Table 42); 
students in evening studies by age and type of school (Table 43). 
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3. FINANCIAL INDICATORS 



Part II of the document, Higher Schools..., lists twenty- seven financial 
indicators (the table numbers indicated in the parentheses refer to Part II of 
Central Statistical Office (GUS, 2000), viz.: 



COSl - 

COS2 - 
EXPl - 

FUNl - 
FUN2 - 

FUN3 - 

FUN4 - 

ICFl - 

INCl - 
INC2 - 
INC3 - 
INC4 - 

INC5 - 

INC6 - 

INC7 - 

INC8 - 

INC9 - 
INC 10 - 
INCH - 
INC12 - 
INC 13 - 
INCH - 



costs and their stmcture by type of school and kind of activity 
(Table 9); 

costs of education per capita hy type of school (Table 18); 
public expenditure on higher education as compared to 1990 
(Table 2); 

special funds of schools by fund and type of school (Table 14); 
percentage of education costs covered from the assistance fund 
for students and the school scholarship fund by type of school 
(Table 15); 

input-output management of the assistance fund for students by 
type of school and kind of assistance (Table 16); 
distribution of assistance funds for students by type of school 
and kind of assistance (Table 17); 

income, costs, and the financial outcomes of schools by type of 
school (Table 4); 

operational income of schools by type of school Table 5); 
operational income structure by type of school (Table 5. 1); 
operational income structure by kind of activity (Table 5.2); 
operational income structure by type of school - for State and 
non-State schools separately (Table 5.3); 

school income from teaching activities by type of school and 
source of income (Table 6); 

structure of school income from teaching activity by type of 
school (Table 6.1); 

structure of school income from teaching activity by source of 
income (Table 6.2); 

structure of school income from teaching activity by type of 
school - for State schools and non-State schools, taken 
separately (Table 6.3); 

school income from teaching activity by type of school and source 
of income (Table 7); 

structure of school income from teaching activity by type of 
school (Table 7.1); 

structure of school income from research activity by source of 
income (Table 7.2); 

structure of school income from research activity by type of school 
- for State schools and non-State schools separately (Table 7.3); 
school income and costs by type of school and kind of activity 
(Table 8); 

income and costs stmcture by type of school (Table 8. 1); 
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INVl 

INV2 

RESl 

RES2 

RES3 



investment outlays in higher education as compared to 1990 
(Table 3); 

investments and renovation costs in schools by type of school 
(Table 13); 

gross financial result of schools by type of school (Table 10); 
charges on gross financial result of schools by type of school and 
kind of charge (Table 11); 

net financial result of schools by type of school (Table 12). 



4. POLISH INDICATORS VERSUS WORLD DECLARATION OBJECTIVES 

The World Conference on Higher Education, Higher Education in the 
Twenty-First Century: Vision and Action (Paris, October 5-9, 1998) confirmed 
the need for the renewal and the re -orientation of higher education at the 
system and institutional levels. The two major documents of the Conference 
- Higher Education for the Twenty first Century: Vision and Action (the so- 
called World Declaration) (UNESCO, 1998a), and Framework for Priority 
Action for Change and Development of Higher Education (the so-called 
Priority Action Plan) (UNESCO, 1998b) - proposed the conceptual 
framework, the main directions of the reform, and their governing 
principles. The World Declaration, in particular, defined four major 
objectives, as follows: 

- Promotion of access to higher education based on merit and addressing 
the areas of equal opportunity; seamless and open access to higher 
education and assistance to minorities and the disadvantaged. 

- Modernization of higher education systems with a focus on access 
rates and overall participation rates; funding for research and 
development, and review of international trends, including fees and 
mobility trends. 

- Renovation of institutions of higher education in the areas of academic 
quality, curriculum relevance, research links and performance, staff 
assessment, and international co-operation. 

- Closer linkages with society; lifelong sources of professional training 
and the development of entrepreneurial skills. 

The above objectives were used by the authors of a report (Abercromby 
and Fielden, 2000) to design a guide for UNESCO Member States to test the 
effectiveness of their higher education policies at both system and 
institutional levels. Taking into account that “governance and management 
arrangements differ among institutions and countries, depending on factors 
such as size, mission, and culture”, they suggested a set of principles and of 
self-challenge questions that can be applied to all institutions and systems. 
The authors also gave an example, in the Annex of their report, that showed 
how the commonly used indicators could be assigned to the UNESCO 
framework for higher education. That example has been used here for the 
assessment of the sets of indicators listed in Sections II and III. The result of 
this assessment is shown in Table 1 . 
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Table 1. The correspondence of the indicators published by the Central Statistical 
Office of Poland (GUS, 2000) with the objectives defined in the UNESCO World 
Declaration (UNESCO, 1998a). 



GOALS 



OBJECTIVES 



INDICATORS 



Equality of access to 
institutions of higher education 
regardless of race, sex, 
language, religion, age, 
economic, and social 
distinctions, and/or physical 
disabilities. 

To gauge the strength of the 
link between higher education 
institutions and research. 

To gain an indication of the role 
of higher education as a catalyst 
for tlie entire system of 
education. 

Ease of choice and flexibility of 
entiy and exit points to 
institutions of higher education. 



To strengthen the link between 
higher education and research 
institutions. 

To gain an indication of tlie level 
of collaboration between higher 
education institutions and 
different sectors of society; to 
ensure that higher education 
and research contribute to local, 
regional, and national 
development. 

Fulfillment of human, material, 
and financial commitments 
made to higher education in 
particular, by the state, over the 
past decade. 

To measure the participation 
and decision-making roles of 
women at all levels and in all 
disciplines in higher education. 
To gain an indication of the 
national and international 
mobility of the teaching staff 
and students. 

To measure the degree of 
autonomy awarded to 
institutions of higher education 
and research in order to fulfill 
their obligations to society. 



Promotion of access based 
on merit (Social - equal 
opportunity; economic 
assistance to minorities 
and the disadvantaged) 

Modernization of Systems 
(Funding - higher 
education research). 

Modernization of Systems 
(Higher education funding 
and overall participation 
rates). 

Promotion of access based 
on merit 

(Cultural - seamless and 
open access to higher 
education). 

Institutional effectiveness 
(Research links and 
performance) 

Institutional effectiveness 
(Research links and 
perfoiTnance) 



Modernization of systems 
(Human development and 
higher education funding) 



Promotion of access based 
on merit (Social - equal 
opportunity) 

Modernization of systems 
(Mobility trends) 



Institutional effectiveness 
(Autonomy and 
perfonnance) 



Indicators portraying the degree 
of access to higher education: 
AlDl-2, CONI, STFl-6, STHl-3, 
STUl-16, EXPl, FUNl-4 



Indicators portrajdng the links 
between higher education 
institutions and research include: 
SCll-9, INC3-4, INC9-12 
Indicators portra}dng the catalytic 
impact of higher education on the 
entire education system: STU2, 
STU4, STU6, EXPl 
Indicators portraying student 
enrollment rates relating to the 
type and mode of tertiary 
institutions: STUl-16 

Indicators portra}dng the links 
between higher education and 
research: EMPl, SCI 1-9, COSl, 
INC3, INC9-12 
Indicators showing the 
collaboration in R&D: EMPl, 
INC3, INC9-12 



Indicators showing whether or 
not financial /human 
commitments made to higher 
education have been fulfilled 
since 1990: EXPl 
Indicators portraying change in 
distribution by sex in higher 
education: EMPl, GRAl-9, GRFl- 
4, STUl-16 

Indicators portraying the degree 
of international student mobility: 
STFl-6 

Indicators portraying the funding 
sources for research and 
development: ICFl, INC 1-14, 
INVl-2, I^Sl-3 
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GOALS OBJECTIVES 

To raise enrollment levels and to Promotion of access based 
expand the level of access to on merit (Economic 
higher education of minorities assistance to minorities 
and the disadvantaged in the and the disadvantaged), 

public and the private sectors. 



To provide complementary 
training to enter the world of 
work from secondary education 
and to facilitate a range of 
gateways for persons wishing to 
gain entry into higher education 
institutions. 

Higher education institutions 
establish and foster their 
missions and degree courses, 
which contribute to the efforts 
of regions to reach an 
environmentally sustainable 
level of economic and social 
development and cultural 
creativity. 

How well higher education 
institutions abide by the rules 
ethics and scientific and 
intellectual rigour, and a 
multidisciplinary and trans- 
disciplinary approach. 

How well institutions have 
established systems of access 
for the benefit of all persons 
who have the necessary abilities 
and motivations to pursue 
higher education. 

To measure how well higher 
education institutions 
conbibute to the sustainable 
development of society through 
the analysis of emerging social, 
cultural, economic, and political 
trends. 

To measure how well university 
faculty members participate in 
teaching, resccirch, tutoring of 
students, and steering of 
institutional affairs. 



Society and work (Lifelong 
source of professional 
training). 



InsUtutional effectiveness 
(Cuniculum and course 
relevance). 



Institutional effectiveness 
of (Quality) 



Institutional effecbveness 
(Participation and access) 



Institutional effecbveness 
(Qutrlity; curriculum; 
research performance) 



Institutional effectiveness 
(Staff participation and 
assessment) 



To gauge how well higher Institutional effectiveness 

education institutions are (Curriculum and course 

serving the community, through relevance; resecirch links 
bieir approach to the analysis of and performance; ethos), 
challenges, problems, and 
different subjects. 



INDICATORS 

Indicators illustrabng the shifts in 
enrollments in higher educaUon 
over time and public and private 
financial support for 
minoribes/disadvantaged 
students: EXPl, FUN 1-4 
Indicators portraying the shifts in 
the percentages and the numbers 
of students enrolled in different 
types of tertiary institutions: 
CONI, STU2, STU4, STU6 for 
1995-1999 

Indicators revealing the shifts in 
the numbers of graduates over 
time, by study field: GRA4-10, 

SCI 1-6 



Indicators showing the origins of 
R&D funds and where these 
funds are then distributed: EMPl, 
INC9-12 



Indicators showing the success of 
systems of access: CONI 



Indicators portraying the 
numbers of persons undertaking 
research: EMPl 



Indicators portraying stalf 
breakdown: EMPl 



Indicators showing how 
institutions of higher education 
serve their communities through 
the analysis of challenges, 
problems, and different subjects: 
GRAl-10, SCI4-7,STU7, STU9, 
STUll, STU13 
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GOALS 



OBJECTIVES 



INDICATORS 



To measure the promotion and 
the development of research in 
all disciplines by the higher 
education institution. 

To measure higher education 
institutional programmes 
aiming at removing inequalities 
related to sex in curricula and 
research and at balanced 
representation among all levels 
of management. 

To gain an indication of the 
student support offerings of 
higher education institutions. 
How well governments and 
industiy support closer links 
l^etween higher education and 
the world of work, so as to 
facilitate the employability of 
graduates who will increasingly 
be required not only to be job 
seekers but also to become job- 
creators. 

How well higher education 
institutions are open to adult 
learners through different 
contexts and ensuring that 
credit is transferable within and 
between institutions, sectors, 
and states. 

To measure how well higher 
education institutions are 
setting up partnerships and 
systems as a means to bridge 
the gap between rich and poor 
countries in the vital areas of 
knowledge production and 

application. 

Source: The author. 



Institutional effectiveness 
(Research links and 
performance) 

Promotion of access based 
on merit (Social - equal 
opportunity). 



Promotion of Access on 
Merit (Social - equal 
opportunity). 

Society and Work 
(Development of 
entrepreneurial skills - 
higher education to turn 
out job creators and social 
responsibility.) 



Promotion of Access on 
Merit (Cultural - seamless 
and open access to higher 
education) 



Institutional Effectiveness 
(International co-operation 
activities) 



Indicators portraying the 
discipline of higher education: 
STUl-16 

Indicators indicating the 
breakdown by sex in tertiary 
institutions: GRAl-10, STUl-16 



Indicators showing the type and 
category of student support: 
AIDl-2, FUNl-4 

Indicators portraying the sources 
of funds (government, productive 
enterprise, foreign, and other 
funds) to fund research and 
development INC9-12 



Indicators showing the extent to 
which education institutions are 
open to adult learners: CONI 



Indicators portraying the 
distribution of foreign students by 
countiy, region, etc.: STF6 



The Central Statistical Office (GUS, 2000) contains very limited 
information on the research activities of higher education institutions in 
Poland. For the time being, no statistical document is being published by 
the Central Statistical Office that systematically presents - year by year, in 
the same logical framework - indicators relative to research and 
development. Such indicators have been available, since 1994 in the 
statistical bulletins published twice a year by the Directorate for Science, 
Technology, and Industry of OECD under the title. Main Science and 
Technology Indicators. Unfortunately, these bulletins are not widely used by 
decision-makers and managers of higher education institutions in Poland. 
The situation may change soon because the Central Statistical Office is 
going to publish, on a regular basis, a bulletin entitled Main Science and 
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Technology Indicators that will be based on the same system of indicators as 
those recommended by OECD, in the Frascati Manual (OECD, 1993). A 
dictionary of the definitions of concepts relative to the statistics of science 
and technology has already been published (GUS, 1999). 

5. COMMENTS AND CONCLUSIONS 

A comparison of the results of the analysis portrayed in Table 1 with the list 
of indicators that are currently available, recommended for data collection, 
and stated in the Annex to Abercromby and Fielden (2000), reveals the 
enormous amount of work that must be accomplished by the Polish 
statistical services to meet UNESCO expectations. This amount of work 
turns out, in practice, to be even greater than estimated on the basis of this 
comparison, owing to the observation that the definition of the indicators 
already being recorded requires serious revision. An overview of some of the 
relevant problems follows. 

First, some of the attributes of a student, that were indisputable in the 
past, have become vague or fuzzy given the recent changes in the systems of 
education. In a so-called flexible study system, a student is allowed to 
define individually not only his or her programme of study but also the pace 
of advancement in his or her course programme. Consequently, the 
assignment of the student to a semester or year of study is becoming 
problematic; what can be reliably recorded by the time of registration is the 
percentage of the programme requirements satisfied. The quantification of 
this indicator is much more accurate than the quantification of semesters 
or years of study completed. 

Another problem is related to the fact that undergraduate students are 
permitted, under certain conditions, to skip the BSc diploma and to work 
for the MSc diploma in order to earn it rapidly. It is difficult to say whether 
they are graduate or undergraduate students, e.g., a month before receiving 
a Master’s degree. 

If the academic staff is considered, numerous statistical problems result 
from the diversification of the teaching load and from the simultaneous 
employment of Polish professors in various schools. The statistics based on 
the numbers of full-time and part-time employees may turn out to be veiy 
unreliable for many reasons: 

- Unlike in the past, two part-time positions are not equivalent to one 
full-time position, but to various possibilities ranging from 0.2 to 1.8; 

- The teaching loads of two persons having identical academic positions 
may vaiy as much as 1:3 owing to the extra paid hours of work 
undertaken by one of them; 

- A university professor may be employed in several schools and 
counted by each of them as its employee. 

The most serious methodological problems appear when trying to 
estimate correctly the total costs of education per capita. There are several 
sources of ambiguity: 
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- The indicator, “costs of education per student per year”, seems to be 
outdated as it does not fit the reality of flexible and diversified systems 
of study. The cost of the whole cycle of education leading to a diploma 
would be a viable alternative; 

- The costs related to the salaries of university teaching staff should be 
divided among their principal activities: teaching and research. There 
are no clear principles that would justify any proportion for such a 
division. 

- The same applies to the costs of administration, investments, and 
modernization; the costs of laboratory equipment and literature; etc.; 

- The links between teaching and research are particularly tight at the 
graduate level; consequently, the problem of separating costs is 
becoming even more complicated. 

Taking the difficulties identified above into account, one should think 
about priorities rather than about the whole far-reaching programme of the 
design and use of indicators. Simple indicators related to students, staff, 
and school finances should first undergo careful revision and discussion, 
both at national and international level. Such an approach would serve as a 
protection against the elaboration of a devastating bureaucracy. 

The last (but not the least problem) to be considered seriously is that of 
indicator uncertainty. From an engineering point of view, this problem is a 
standard one in the science of measurements, but one that is 
underestimated or overlooked by the statisticians of education; re., evidence 
in the Central Statistical Office (GUS, 2000) and OECD (2000). 

According to the engineering art, the design of a system of effective 
management in higher education requires reliable information about its 
state - reliable does not necessarily mean certain, but rather uncertain, 
with a known level of uncertainty. Eveiy year, when the author’s institution, 
the Warsaw Technological University, prepares its data for the Central 
Statistical Office, it must deal with numerous ambiguities by the taking of 
arbitrary decisions. Probably all institutions follow the same pattern, and 
one can assume that an accumulation of such decisions may produce 
statistically significant by-effects. But there are no instruments in use to 
identify or to quantify these. Consequently, the final documents do not 
contain any information on the uncertainty of indicators. 
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IX. Romanian Perspectives on the Design and Use 
of System-Level Indicators in Higher Education 



PANATTE NIC A 



1. INTRODUCTION 

This study analyzes the design and implementation of system-level 
indicators for higher education from the Romanian perspective. For each of 
the relevant domains {Le., institutions, administration, perfonnance 
evaluation, financing, and management), the major evaluation indicators 
are identified and the compatibility between these indicators and those of 
UNESCO is also discussed. 

A new set of evaluation indicators for use at institutional level is 
proposed. It could be used for the ranking of Romanian universities. It has 
been partially tested in Romania, and the results have been made available 
to the universities concerned. An analysis of the overall design of the 
UNESCO System-Level Indicators for Higher /Tertiary Education is presented. 

2. CONSIDERATIONS REGARDING THE DESIGN OF SYSTEM-LEVEL 
INDICATORS FOR HIGHER EDUCATION 

The indicators for the evaluation of higher education employed in Romania, 
like those used in other countries, are strongly determined by the 
characteristics and particularities of the respective higher education system 
and the way in which it is organized. 

The UNESCO evaluation indicators for higher education, like any 
international statistics, can be assessed according to the following 
fundamental criteria: 

L Their elaboration of a perception of the status [Le., the level for any 
given year) and the dynamics {Le., the trend of changes over a given 
period) of, and environmental factors {Le., international, regional, 
national, or local) regarding the implementation of the vision of higher 
education as formulated in Higher Education in the Twenty-First 
Century (UNESCO, 1998a), and the Framework for Priority Action.... 
(UNESCO, 1998b). 

iL Their setting of a level of data compatibility by the definition of each 
particular indicator as precisely and as accurately as possible in order 
to avoid confusion or improper use. 

UL Their fixing of a relevant degree of indicators for different categories of 
users. A preliminary list of such categories of users could be UNESCO 
and other international institutions {Le., the World Bank, OECD, the 
European Commission), national institutions, universities, and other 
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higher education institutions in each country, and, last but not least, 
the individuals directly involved in the processes of higher education,. 
Le., professors and students. 

iv. The extent to which they aggregate at each of the following two levels: 

- at the level of dfferent areas of applicability, Le., at the university, 
country, and regional levels (considering not only geographical 
location, but also the level of development, etc); 

- in terms of primary indicators which measure a single aspect with 
all-encompassing indicators referring to more general phenomena 
This type of aggregation should be achieved based on a model of 
^SS^^S^tion which allows for and takes into account the possible 
existence of different priorities, which may change over time, or 
various degrees of fulfillment of set objectives. 

The author believes that in designing a system of evaluation indicators for 
higher education, the usability of the information inherent to such a 
system, assessed in terms of its relevance for strategic decisions to be taken 
by the users of the system, should constitute a major factor. 

Therefore, special attention should be paid to identifying: 

- the categories of users of the system of indicators: UNESCO and other 
international institutions [Le., The World Bank, OECD, the European 
Commission), national institutions, universities, and other higher 
education institutions in each country, professors, and students: 

- the types of decisions and of the information to be extracted from the 
system of indicators. These will be a function of the mission, the 
objectives, and the responsibilities of the different types of users 
mentioned above. 

3. CHARACTERISTICS OF THE HIGHER EDUCATION SYSTEM OF 
ROMANIA AND THEIR RELATION TO EVALUATION INDICATORS 

3.1. The Organization of the Higher Education System and the Types of 
Educational Programmes 

As an integral part of the Romanian national education system, the higher 
education system consists of both public and private institutions. Higher 
education is provided in education and research institutions - universities, 
academies, conservatories, and academic colleges. 

Higher education institutions may operate legally after their functioning 
has been authorized by the Government and approved by the National 
Council for Academic Evaluation and Accreditation. 

The public higher education sector in Romania includes forty-nine public 
universities which can be classified according to the dominant profile of 
their curriculum: complex profile universities (mainly social-humanistic and 
scientific), technical universities, medical universities, agricultural science 
universities, veterinaiy medicine universities, and ai*t universities. 
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Numerous private education institutions (over eighty universities) have 
been set up as foundations since 1990, covering all forms of education, 
from nursery schools to universities. In order to control this rapid growth, 
the Law on the Accreditation of Higher Education Institutions and Recognition 
of Diplomas was passed in December 1993. 

The most significant difference between public and private higher 
education lies in the ways in which they are respectively funded. Public 
higher education institutions are financed mostly from the state budget. 
Tuition fees and other individual incomes represent a secondary and 
complementary source of financing. 

Private higher education institutions are mainly self-financed: they 
support themselves from tuition fees and sponsorships. Provided they are 
accredited, they are eligible to compete for state budget funds, such as 
development funds, research funds, and post-graduate scholarships. 

Higher education institutions usually include several faculties, colleges, 
and departments. The faculty represents the functional basic unit of the 
higher education institution and is organized in sections and departments. 
The Romanian higher education system offers full-time courses, evening 
courses, part-time courses, and distance learning in the following types of 
programmes (Table 1). 



Table 1 . Types of academic programmes in Romanian higher education institutions 



Level of studies 


Types of academic programmes 


Years 


Diploma 


Level 1: 

Undergraduate 

programmes 


Short-tenn higher education 
programmes 

Long-term higher education 
programmes 


3 

4 - 6 


College Diploma 
University Diploma 


Level 2: 

Postgraduate 

programmes 


Advanced studies 
Master s degree studies 
Further education academic 
courses 

Postgraduate academic studies 
Doctoral studies 


1 

1 - 2 

0.5 to 1 

2 - 3 
4 - 6 


Diploma of Advanced Studies 
Master s Diploma 

Further Education Diploma 

Diploma of Postgraduate Studies 
Doctoral Diploma (PhD) 



Source: The author. 



Rapid growth occurred in the numbers of students during the 1989- 
2000 period, as reflected in the improved perfonnance assessed as based on 
' UNESCO indicators. This process resulted in improvements in terms of 
performance, through an increase in the level of student enrollment in 
higher education. At the same time, however, it also led to the persistence of 
some dysfunctions regarding the distribution of students by specialized 
study fields in the public universities. The discrepancies in the numbers of 
students by profile led to the complete funding of certain specialties from 
the state budget, while others were left almost entirely to the private 
institutions, a result which had negative effects upon the quality of the 
teaching process and the efficient use of the funds budgeted for education. 

In 1990, Romanian education had a non-symmetrical position in 
comparison to that of other European countries, the former socialist 
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countries included. According to UNESCO statistics, the structure of the 
educational profile in Romania continues to be essentially different from . 
that of other European countries - and it basically retains the structural 
characteristics of education in the period prior to 1990. The development of 
private higher education in Romania, especially in the fields of economics, 
law, the social sciences, and medicine, as well as the great discrepancies 
between public and private institutions, reflect, so far as competition on 
entrance examinations is concerned, the contradiction between the number 
of places financed from the state budget and actual social demand for 
higher education. 

3.2. Higher EdiLcation Management 

At national level, for almost half a century, the Romanian education system 
was extremely centralized. After 1989, a flexible higher education policy was 
designed under a new Law on Education (1995). This policy aims at 
strengthening institutional autonomy by decentralizing decision-making 
and the management of higher education. This law is now being 
implemented by the Ministiy of National Education, assisted by advisory 
bodies and in consultation with the national scientific associations to which 
the academic staff members belong, and nationally acknowledged student 
organizations as well as the teaching staff unions. 

The Ministry of National Education initiated the establishment of 
academic advisory bodies (see Figure 1), to which it has delegated 
responsibility for establishing strategies, adopting procedures, and 
monitoring various aspects of education, in accordance with its own 
mission and objectives, as well as with the fundamental principles of higher 
education reform in Romania. The composition of these councils includes 
exclusively academic staff, usually nominated by the most important 
universities, in accordance with their respective competencies. 

In the elaboration of strategies, mles, regulations, and legal initiatives, 
the Ministry of National Education works in close co-operation with 
academic advisory bodies and higher education institutions. The major 
responsibility of both the Ministry and the Councils is the elaboration of 
broad strategies. Decisions are based on a combination of the following 
statistics; 

- international statistics, including UNESCO statistics; 

- official national statistics, prepared by the Romanian National 
Commission for Statistics; 

- the Ministry’s own statistics, as well as those of the Councils; 

- statistics prepared by various education institutions. 

It should be noted, however, that, with the exception of the official 
statistics prepai'ed by the National Commission for Statistics, a unified, 
well-structured, and integrated system for providing information and 
evaluation indicators at national level has not yet been achieved. As a 
consequence, limited access to such information, as well as certain 
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Figure 1 . Responsibilities of higher education institutions in Romania 



INDICATORS FOR DECISION-MAKING AT COUNTRY 
LEVEL RELATIVE TO: 

— Strategic co-ordination of education system 
development 

-- Establishing the institutional competence 
framework by assessing academic performance and 
quality and by allotting funds according to 
perfonnances and to the strategic options for 
national growth 

“* Strategies and principles 

~ General regulations 

~ Academic and administrative strategic decisions 

— Financial and investment planning 

— Prevention of distortions in the higher education 
system 



INDICATORS FOR DECISION- 
MAKING RELATIVE TO: 



MINISTRY OF EDUCATION 



Academic Advisory Bodies 
The National Council of Academic 
Evaluation and Accreditation 
National Rectors' Conference 
National Council for Attestation of 
Academic Titles. Diplomas, and 
Certificates 

National Higher Education 

Funding Council 

National Council for Academic 

Research 

Other 



UNIVERSITY 



UNIV. SENATE 



51 



— Genereil regulations 

— Academic and administrative 


ACADEMIC 


21 




1 

SENATE BOARD 


8 


strategic decisions 


BOARD 






1 , 





planning 



RECTOR 

I 



STRATEGIC 
MANAGEMENT AT 
UNIVERSITY LEVEL 



FACULTY 



Faculti^ Council 
(11 -27) 



Deputy 

Dean 



INDICATORS FOR DECISION- 
MAKING RELATIVE TO: 

— Cumcula 

— Promotions 

— Syllabi 

— Faculty evaluation 

— Major and minor fields 

— Research programmes 

” Graduate and PhD studies 

” Financial and investment 
planning at the faculty level 



ACADEMIC MANAGEMENT AT 
FACULTY LEVEL 



Faculty Board 
(3-7) 



VICE-RECTORS 


4 




CHIEF 




ADMINISTRATIVE 








ACCOUNTANT 




DIRECTOR 






Dean 



L3 



Dept. 



Dept. 



Dept. 



ADMINISTRATIVE 

DEPARTMENTS 



INDICATORS FOR DECISION-MAKING 
RELATIVE TO: 

Maintenance, catering, 
accommodation 

— Staffing 

— Investment implementation 



ADMINISTRATIVE MANAGEMENT 



Source: The author. 



BEST COPY AVAILABLE 



128 



P. NICA 



incompatibilities inherent in the design of indicators, makes the process of 
measurement and evaluation ambiguous and, at times, difficult, which, 
means that an accurate assessment of the real situation cannot always be 
achieved. 

The content of the system of indicators for higher education employed at 
different levels the Ministry, advisory bodies, universities, and faculties) 
was designed so as to convey information which would allow decision 
making based on specific spheres of competence (Figure 1). At the same 
time, such indicators should reveal the connections and relationships 
present both at the horizontal level (among functional domains, ie., the 
educational process, scientific research, academic evaluation and 
accreditation, financing, etc.) and at the vertical level (among structural 
components). 

The indicators corresponding to functional domains, even though 
strongly influenced by the actual content of activities, should be designed to 
convey information that can be applied, directly or indirectly (based on 
certain aggregation models), in other domains as well, without creating a 
need to re-create or to replicate the existing databases. This objective has 
not been accomplished, as yet, in the Romanian higher education system. 

3.3. Performance Evaluation 

The performance evaluation of Romanian higher education institutions 
comes within the responsibility of the National Council for Academic 
Evaluation, Authorization, and Accreditation (NCAEAA). 

According to the Law on Accreditation, all existing higher education 
institutions and all new academic programmes are subject to performance 
evaluation. There are several stages in the evaluation and accreditation 
procedure. 

- Provisional Licensing. Any institution, which offers a new educational 
programme or establishes a new faculty or new college, must apply to 
NCAEAA for provisional licensing. Graduates of provisionally licensed 
programmes must take their final examinations at accredited 
institutions, which can award degrees and diplomas. 

- Accreditation. Accreditation applies to provisionally licensed 
programmes, which meet all the legal criteria and standards during 
the time for which the license has been issued. As soon as a 
university is accredited, it becomes autonomous and is entitled to 
award legally recognized diplomas and certificates in its own name. 

- Periodic Evaluation. The Law provides that all universities (with all 
their faculties, colleges, and programmes) be subject to evaluation 
eveiy five years. 

The initial information is extracted from the self-evaluation reports, 
which provide information on the university as a whole, on the faculty 
(department or college), and on the programmes. The evaluation criteria 
employed in assessing the quality of educational programmes can be 
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of instruction: teaching staff; academic research; infrastructure; financial 
and management activities. 

The self-evaluation reports used as the basis for evaluation include 
statistics regarding the above mentioned indicators for a period of seven to 
nine years. This period reflects the usual duration of studies (four to six 
years) plus three cohorts of graduates, which applies to the specialized 
study fields being submitted for evaluation, as well as to the faculties and 
universities to which the fields belong. 

Since the vast majority of the self-evaluation reports are not submitted in 
electronic format, but rather as hard copy, the bulk of the statistics cannot 
be integrated into a single national system of indicators for higher 
education. 

3.4. Financing 

The system of higher education in Romania is financed by the Ministiy of 
National Education, on the recommendations of the National Council for 
Higher Education Funding (NCHEF). 

A new financial procedure, called Global Financing, was designed and 
implemented starting with the 1998-1999 academic year. According to this 
procedure, budgeted funds are allocated to higher education institutions on 
the basis of an institutional agreement. 

Global Financing has two components: core financing, representing 80 
percent of the entire budget for higher education, and complementaiy 
financing, representing 20 percent of the same budget. Higher education 
institutions receive their core financing based on a formula which considers 
the equivalent number of students, while the complementaiy component is 
allotted on the basis of competition. 

The reform programme for higher education includes substantial 
changes in the field of financial management, ensuring academic autonomy. 
These changes can be summarized as follows: 

- Institutions are autonomous and make their own decisions as to how 
to use the funds received, in accordance with their institutional 
development plans. 

- They have complete financial autonomy regarding the creation and 
the management of their own resources. 

- Institutions have the right to require that their subsidized students 
pay administrative fees. 

- Public institutions are allowed to enroll tuition fee-paying students. 

According to the Romanian Law on Education, 4 percent of GNP should 
be allocated, via the state budget, to the financing of education. 
Unfortunately, this provision has never been implemented. The percentage 
of GNP allocated to education has been considerably lower than that in 
other European countries. On the other hand, the low level of budgeted 
allocations for education, along with the low level of GNP, has caused the 
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public funds available for the financing of education to be much lower than 
in most European countries. 

Public higher education may also be financed from non-govemmental 
resources, both internal and external. Internal sources include services or 
activities, contributions by business organizations, and tuition fees. The 
mean value of the percentage of the individual income of universities (Le., 
generated from both internal and external sources) was 34.94 percent in 
1999, with a variance of 56 percent for the pool of universities considered. 

The essential indicator for the assessment, planning, and allocation of 
funds is the level of the unit cost per equivalent student. This indicator is 
one of the essential components of the package of measures required for the 
implementation of the reform, according to which financial resources 
should meet student needs. 

The unit cost per equivalent student can be thought of as having two 
forms: 

- the overall cost per equivalent student (expenses covered from core 
and complementary financing) ; 

- the net cost per equivalent student (expenses covered only from core 
financing). Tlie net unit cost is an indicator which estimates the 
required amount of core financing at the level of each university 
compared to the whole Romanian higher education system and the 
fair distribution among universities of the funds approved in the 
national budget. 

The general equation for the establishment of a unit cost (be it overall or 
net) (Nica, 1999) is 

Qi/se“^/He’ 



unit cost per equivalent student; 
numbers of equivalent students: 
total expenses. 

The expression, number of equivalent students, is common to both 
indicators mentioned above. For it to be determined at the university level, 
two elements must necessarily be known: (i) the average number of 
participants in each of the forms of education existing in universities, and 
(ill the equivalence coefficient corresponding to each form of education 
stipulated by the National Council for Higher Education Funding. The 
equivalence coefficients have been set by the National Council for Higher 
Education Funding based on the comparison of teaching hours spent in 
different forms of training. This indicator, computed at the level of each 
university and at the national level, enables several comparisons among 






where 

c 



u/se 



N. 



C, 



ROMANIAN PERSPECTIVES 131 



different universities and profiles of education. The number of equivalent 
students is the basic indicator to be considered in the elaboration of the 
annual draft budget, both at the level of each university and at the level of 
the Ministry of National Education. 

The unit cost per equivalent student is influenced by numerous factors, 
among which the design and organization of teaching activities. The 
administrative system plays an essential role. 

Each of the two factors can be measured on the basis of a number of 
indicators, as follows: 

- indicators regarding teaching activities: 

number of specialized study fields and the average number of 

students per study yecir: 

number of teaching hours per week; 

ratio of working hours in small groups to the total of practical work 

and workshop hours; 

average number of students in a group; 

extent to which teaching positions are filled; 

ratio of the numbers of equivalent students to the number of 
teaching jobs. 

- indicators regarding the administrative system: 

numbers of maintenance staff; 
teaching space in m^ per student; 
numbers of students per non -teaching job. 

The value of each of these factors is directly determined by the specific 
characteristics of the educational process in different study fields [e.g., 
engineering, humanistic sciences, medical sciences, etc.); moreover, their 
values can change over time. 

For determining the unit cost per equivalent student, as well as the 
budget fund allocations to universities, a cost coefficient is used. This cost 
coefficient is calculated as the ratio between the unitary cost for different 
specialized fields and the unitary cost for the economic and social- 
humanistic field, which is considered as the basic line. This indicator is 
employed in determining other indicators, which measure both the 
financing process and the performance evaluation regarding the financial 
management at institutional level. 

In conclusion, the following major problems can be identified: 

- If the indicators refer to the measurement of certain aspects of the 
higher education system such as trends in the numbers of students, 
either globally or according to certain structural groupings (sex, study 
field, education, forms of education, etc.), the level of enrollment in 
higher education, or the numbers of students per 100,000 
inhabitants, the physical numbers of students should be taken into 
account. 
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- If the indicators refer to public as well as to private financing of higher 
education, based on such factors as expenses per student, expenses- 
from public and private funds, etc., the use of the above indicator (Le., 
the physical number of students) is not appropriate, since it leads to 
improper comparisons and therefore could be irrelevant. The values of 
these indicators are strongly determined by the student distribution 
per study field. In this case, it is necessary to use indicators that take 
into account the number of equivalent students. However, such an 
indicator - or any other, which would allow a valid basis for 
comparison - has not been taken into account in the compilation of 
international statistics. 

- In some countries, Romania included, only certain categories of 
students or specialized study fields have access to public funds. As a 
result, there will be significant differences across countries, which will 
affect the comparability of statistics coming from such countries. For 
instance, in Romania, only public universities have access to public 
funds for the financing of higher education. But even in the case of 
public universities, there is a distinction between students financed 
from the state budget and self-financed students (paying tuition fees). 
The public universities do not receive budgeted funds for the last 
category of students, who, for all practical purposes, have the same 
status as students in private universities. At the same time, certain 
forms of higher education [e.g., distance learning) or postgraduate 
studies are funded only from private funds {Le., fully financed through 
tuition fees). 

- To ensure the comparability of statistical data among different 
countries when determining such indicators as that of “public funding 
per student”, the number of equivalent students must be taken into 
account and can only be calculated on the basis of the number of 
students who can actually be financed from such funds. Otherwise, 
the indicator itself slides into irrelevance. 

4. PROSPECTS FOR THE UTILIZATION IN ROMANIA OF A SYSTEM OF 
PERFORMANCE EVALUATION INDICATORS FOR HIGHER EDUCATION AT 
INSTITUTIONAL LEVEL 

At present, Romanian universities are undergoing a profound 
transformation so as to align themselves with international trends and 
standards in higher education. They are continuously improving in terms of 
meeting their academic objectives, while their activities continue to be 
shaped by social and economic realities. Autonomous decision-making, at 
academic and financial levels, allows them to adopt distinct management 
strategies, with objectives specific to each university. At the same time, 
experience makes a difference in terms of the degree to which universities 
undertake their major tasks, offering instruction and undertaking research 
of high quality. 
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Cun*ently, various judgments regarding the level of performance of the 
educational process in universities are appearing from various sources, but 
the arguments that fail to be substantiated are often subjective. At the same 
time, the Law of Education specifies that 

core funding is budgeted by the State according to the number of 
undergraduates and PhD students attending tuition-free courses, 
lecture participants, level and nature of the theoretical and practical 
training. Other specific indicators related to educational activity, 
especially when referring to the quality of teaching, are also taken into 
consideration (Government of Romania, 1995, Art. 171, it. 3). 

The indicators cited above, however, have not yet been formally specified. 
Consequently, the current funding of universities is non-differential: 
funding decisions disregard differences in the efficient use, by universities, 
of budgeted allocations. 

In some European countries, specific comparative procedures have been 
implemented in order to assess university performance. They aim either at 
the separate assessment of teaching and research or at the assessment of 
both activities combined by means of a synthetic indicator. 

Among the current procedures, and taking into consideration both the 
synthetic features of the assessment and the given experience, the author 
considers that the quality award-based assessment process represents a 
thorough assessment of the different aspects related to the performance of 
universities at institutional level. 

Since 1999, the Romanian Ministry of National Education, supported by 
a PHARE grant, has been working on the design of a system of indicators for 
the performance evaluation of higher education at institutional level in 
Romania (Nica, 2000). 

After an analysis of the concept of performance at institutional level, the 
following seven main dimensions were chosen: 

L Academic prestige; 

a. Selectivity by students and university attractiveness; 
at Management of human resources; 

iv. Scientific research, continuing education, and Master’s and Doctoral 
Degrees; 

V. Student and graduate performances; 

vL Financial resources and ability to provide needed teaching conditions; 
uii. University strategic management. 

Each of these dimensions (synthetic indicators) of the system of 
indicators consists of additional analytical indicators, each one having a 
specific method of calculation. 

As far as some of the indicators are concerned, the data required in 
detenrrining their value can be obtained from existing reports by the 
Ministry of Education (statistical reports on the teaching structures and the 
teaching staff, or the number of students, institutional agreements. 
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accounting audits, etc). For the rest of the indicators, the data can be 
obtained from the national councils mentioned above. Universities 
themselves can also supply the information needed to determine the value 
of those indicators for which no other sources of information are available. 

Some indicators require the development of new activities that the 
universities have not undertaken or have only started to undertake. These 
include student services [ie., career development and assistance in job 
searches, adaptation of the curriculum to market-required skills, the 
development of quality management systems, etc.) . 

As those universities featuring a complex configuration would educate 
students in two or more fundamental fields, the global value of a university 
indicator pertaining to them should be calculated as a composite indicator 
by factoring in the weight of each fundamental field according to the 
structure of the mass of students. The process of evaluation could therefore 
apply to similar universities and faculties, and respectively, to students 
belonging to similar fields. By the same token, an evaluation of different 
faculties belonging to the same university might also be undertaken: hence, 
the difference between those faculties helping to push the respective 
university to a top position and those diminishing its qualitative level might 
be ascertained. 

5. CONCLUSIONS REGARDING THE DESIGN OF THE UNESCO SYSTEM- 
LEVEL INDICATORS FOR HIGHER EDUCATION 

Based on the Abercromby and Fielden (2000) study, several changes are 
suggested. 

Considering the vision of higher education as formulated in Higher 
Education in the Twenty-First Century.... (UNESCO, 1998a) and the 
Framework for Priority Action.... (UNESCO, 1998b), the basic proposals 
regarding adjustments in the system of indicators refer to the following 
factors: 

- the organization of indicators in a tree structure which would permit 
their grouping into categories defining distinct aspects of higher 
education at the national or institutional level; 

- the regrouping of certain indicators and groups of indicators on the 
basis of their belonging to specific aspects of the educational process, 
thus ensuring a stricter definition of these indicators; 

- the addition of certain indicators able to define important aspects of 
the educational process at national and institutional levels; 

- the inclusion of certain indicators that could serve as the basis for the 
comparative analysis of institutional performance at the national, 
regional, and international levels; 

- the regrouping of all indicators regarding the effects of actions by 
UNESCO and other international organizations and institutions in a 
separate chapter so as to ensure a coherent and complete vision of 
these actions and effects. 
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We propose the following structure for the UNESCO System-Level 
Indicators for Higher Education: 

1. Promotion of access on merit 

1.1. Social - equal opportunity and access to higher education 

1.2. Cultural - seamless and open access to higher education 

2. Modernization of systems 

2.1. Technological access 

2.2. The contributions of government, local communities, employers, 
and other institutions, at the national and local level, to higher 
education 

2.3. Mobility trends 

3. Institutional effectiveness 

3.1. Institutional quality, accountability, and international standing 

3.2. Scientific and intellectual integrity and ethics in higher 
education 

3.3. Staff development, participation, and human resources 
management 

3.4. Strategic management, a quality management system, and 
policy preparation 

3.5. Curricular and course relevance 

3.6. Research links and performance 

3.7. International co-operation activities 

4. Society and work 

4.1. Lifelong source of professional training 

4.2. Development of entrepreneurial skills 

5. The contribution of UNESCO and other international organizations to 
promoting higher education in the context of globalization. 

In addition, the data led to the following set of proposed changes: 

- the need to use complementary methods to display information, using 
tables as well as charts, so that information is conveyed in a 
suggestive form for users: 

- the need to use a standard monetary denomination for the financial 
indicators, instead of national denominations, to ensure data 
comparability: 

- the need to use time series to illustrate the degree and direction of 
changes. For instance: 

monitoring the trends of a given phenomenon in one country, in 
comparison with the trends in a group of countries, which are 
comparable from the point of view of development level or are 
located within the same geographic area: 

monitoring the trends of a given phenomenon in comparison with 
the trends of another phenomenon, which is related in some way 
with the first [e.g., expenditure per student in higher education as 
compared to expenditure per student in other forms of education, 
or compared to the GNP per capita, etc): 
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- the need to make UNESCO statistics available in a form accessible to 
most users so as to enable further analysis of the data with a 
minimum investment of time and the least possible error. We strongly 
recommend that the data be maintained on Microsoft Excel 
spreadsheets and be made available on the Internet for easy and 
universal access; 

- the design of the presentation layout for the system of indicators in a 
Web format, with hyperlinks from each of the indicators to 
corresponding definitions, databases, and charts. 
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X. A South African Perspective on System-Level 
Indicators for Higher Education 

SALEEMBADAT 



1. INTRODUCTION 

The aim of the 1998 UNESCO World Conference on higher education was 
“to lay down the fundamental principles for the in-depth reform of higher 
education systems throughout the world with a view to strengthening their 
contribution to the building of peace, founded on a process of development 
based on equity, justice, solidarity, and liberty”. To this effect, the 
Conference issued two documents: Higher Education in the Twenty-First 
Century: Vision and Action, and Framework for Prionty Action for Change and 
Development of Higher Education [UNESCO, 1998a; UNESCO, 1998b). The 
documents indicate the goals and objectives to be pursued by higher 
education and the actions to be taken in this respect. 

An important issue is how it will be possible to know whether or not 
countries are indeed making progress towards the achievement of the goals 
and objectives that have been defined and whether actions are being taken 
towards their realization. The Invitational Roundtable on System-Level 
Statistical Indicators for Higher /Tertiaiy Education, that was jointly 
organized by the Research Institute for Higher Education of Hiroshima 
University and UNESCO-CEPES, sought to address this precise issue. 

To facilitate discussion during the Roundtable, a concept paper, 
“System-Level Indicators for Higher/Tertiaiy Education: Some Notes on 
Requirements and Use” was circulated (see Kaiser, 2003, in this volume, pp. 
3 1 -35) . The paper seiwed as a useful entry point into the theme of system- 
level indicators. However, given its purpose, it also gave rise to a number of 
questions that must be addressed if countries are indeed going to be in a 
position to produce the system-level indicators that are being sought. 

The concern of this article is principally the conceptual, strategic, and 
empirical issues that will need to be confronted. However, it also addr'esses 
the past and present situations with respect to system-level indicators in 
South Africa and the likely future and accompan 3 ing issues and problems 
so far as system-level indicators are concerned against a backdrop of the 
stnicture, purposes, and policy goals of South African higher education. 

2. THE SOUTH AFRICAN HIGHER EDUCATION SYSTEM 

The 1996 Constitution of the Republic of South Africa defines higher 
education as a national government competency, as opposed to a provincial 
competency. As a result, higher educgtiorl provision falls under the 
jurisdiction of the national Ministry of Education. The Higher Education Act 
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of 1997 provides the legislative basis and framework for South African 
higher education. 

The higher education sector comprises public institutions - universities, 
technikons (a kind of South African technical college), colleges of education, 
and agricultural and nursing colleges - as well as numerous, generally 
small, private institutions providing higher education. A programme-based 
definition of higher education rather than a purely institutional definition 
means that higher education programmes may also be offered by further 
education institutions. The White Paper of 1997 states that the colleges will 
be incorporated into the higher education sector in phases, beginning with 
the colleges of education. 

There are twenty-one public universities and fifteen public technikons, 
the student enrollments of which, during 2000, were 386,000 and 199,000, 
respectively. During the pre-1994 apartheid period, there were 120 colleges 
of education. Their numbers have been gradually reduced and, during 
2001, all education colleges were incorporated into universities and 
technikons. There are also twenty-four nursing colleges (6,647 students 
enrolled in 2000) and eleven agricultural colleges (2,033 students enrolled 
in 1999), which presently exist under provincial rather than national 
jurisdiction. 

Alongside the public higher education sector, there exists a small but 
growing private higher education sector. The 1996 ConsUiutLon provides for 
such institutions on condition that they do not discriminate on the grounds 
of race, and that they register with the state and maintain standards that 
are not inferior to those at comparable public institutions. The Higher 
Education Act stipulates the legal conditions for the registration of private 
higher education institutions and imposes various obligations. A regulatory 
framework is emerging to ensure that only those private institutions with 
the necessary infrastructure and resources to provide and sustain quality 
higher education will be registered. 

There is a diverse range of key national actors in higher education and 
higher education policy-making. The national Ministry and Department of 
Education regulate higher education provision and attempt to steer higher 
education to contribute to national policy goals through the instruments of 
national planning and public funding. The Council on Higher Education 
serves as the advisory body to the Minister of Education and is also 
responsible for quality assurance (programme accreditation, programme 
reviews, institutional audits, and quality promotion). Umbrella interest 
groups such as the South African University Vice-Chancellors’ Association, 
the Committee of Technikon Principals, the Association of Principals of 
Agricultural Colleges, and the Alliance of Private Providers of Education, 
Training, and Development exist alongside numerous national student 
organizations, labour unions, and research and development agencies. 

As characteristic of periods of social transition. South African higher 
education is in considerable flux. As with other areas of South African 
society, a comprehensive agenda for transforming higher education has 
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been developed. This agenda has its source in at least three related 
conditions. 

First, the inherited higher education system was designed, in the main, 
to reproduce, through teaching and research, white privilege and black 
subordination in all spheres of society, and all institutions were, in varying 
ways and to varying extents, deeply implicated in this purpose. A lack of 
vision, a paralysis of policy, and a lack of legitimacy and conflict around 
governance characterized higher education. It was fragmented and divided 
along racial and ethnic lines, and reflected severe social inequalities of race 
and sex, with respect to student access and success and the composition of 
the academic staff- Finally, there were major institutional inequities 
between what has been termed historically white institutions and 
historically black institutions. Thus, one key policy imperative and 
challenge is to transform higher education so that it becomes more socially 
equitable internally and promotes social equity more generally. 

Second, research and teaching were extensively shaped by the socio- 
economic and political priorities of the apartheid separate development 
programme. Instead, higher education is now called upon to address and to 
become responsive to the development needs of a democratic South Africa. 
These needs are crystallized in the White Paper on the Reconstruction and 
Development Programme (RDP) (Government of the Republic of South Africa, 
1994, as a fourfold commitment. First is “meeting basic needs of people - 
jobs, land, housing, water, electricity, telecommunications, transport, a 
clean and healthy environment, nutrition, health care, and social welfare”. 
Second is “developing our human resources”, defined as cmcial to the RDP 
as a “people-centered programme”. 

Third comes “building the economy”, and finally, the task of 
“democratizing the state and society”, on the grounds that “without 
thoroughgoing democratization, the resources and potential of our country 
and people will not be available for a coherent programme of reconstruction 
and development”. 

Finally, the transition of South Africa occurs during a period that has 
witnessed the emergence of a global economy and changes in the world that 
have been captured by the “globalization” concept. It is recognized that 
economic growth, in the words of Martin Camoy, is “increasingly dependent 
on knowledge and information applied to production, and this knowledge is 
increasingly science-based” (Camoy, 1998, p. 2). Moreover, if the argument 
of Manuel Castells that “if knowledge is the electricity of the new 
informational international economy, then institutions of higher education 
are the power sources on which a new development process must rely” 
(cited, in, National Commission on Higher Education, 1996) enjoys broad 
acceptance (see Manuel Castells’ biography at http: //www. sociology. 
berkeley.edu/faculty/castells), a concomitant challenge facing higher 
education is to produce, through research and teaching and learning 
programmes, the knowledge and people-power that will enable South Africa 
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to engage proactively with and to participate in a highly competitive global 
economy. 

Higher education policy development - from the National Commission on 
Higher Education of 1996, through to the Higher Education Act of 1997 and 
the 1997 White Paper, A Programme for the Transformation of Higher 
Education in South Africa - has taken as its point of departure the triple 
challenge indicated above. This challenge is to overcome social-structural 
inequities, to contribute to reconstruction and development, and to position 
South Africa to engage effectively with globalization. The gravity, the 
enormity of the challenge, becomes more evident when one understands 
that for economic, social, and political reasons, there is no option to 
postpone one or other elements of the challenge or to tackle them in 
sequence. Rather, they must be confronted simultaneously. 

As the White Paper notes: 

[T]he South African economy is confronted with the formidable 
challenge of integrating itself into the competitive arena of 
international production and fincince.... 

Simultaneously, the nation is confronted with the challenge of 
reconstructing domestic social and economic relations to eradicate 
and redress the inequitable patterns of ownership, wealth, and social 
cind economic practices that were shaped by segregation and 
apartheid (emphasis added). 

In relation to this many-headed challenge, the White Paper identifies 
various and, indeed, diverse social purposes for South African higher 
education: 

...attention to the pressing local, regional, and national needs of South 
African society and to the problems and challenges of the broader 
African context. 

...the mobilization of human talent and potential through lifelong 
leaiming to contribute to the social, economic, cultural, and 
intellectual life of a rapidly changing society. 

...to help lay the foundations of a critical civil society, wjth a culture of 
public debate and tolerance which accommodates differences and 
competing interests. 

...the training and provision of people-power to strengthen this 
country’s enterprises, services, cind infrastructure. This [task] requires 
the development of professionals and knowledge workers with globally 
equivalent skills, but who are socially responsible and conscious of 
their roles in contributing to the national development effort and to 
social transfoiTnation. 

Production, acquisition, and application of new knowledge: ...a well- 
organized, vibrant research and development system which integrates 
the research and training capacity of higher education with the needs 
of industry and of social reconstruction. 
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In terms of specific policy goals and initiatives, the higher education 
transformation agenda is constituted by: 

- the development of a single, integrated, national, co-ordinated, yet 
differentiated, and diverse higher education system; 

- co-operative governance of the system and institutions and 
partnerships; 

- increased and broadened participation within higher education to 
meet people-power needs and to advance social equity; 

- curricular restructuring and knowledge production which is 
responsive to societal interests and needs; 

- quality assurance through assessment and promotion of quality and 
accreditation of programmes; 

- incorporation of higher education programmes and qualilications 
within a National Qualifications Framework, designed to promote 
articulation, mobility, and transferability; 

- improved institutional planning and management and the 
development of three-year institutional plans; 

- state funding on the basis of allocated student enrollments and 
accredited programmes with redress funding to overcome historical 
institutional inequities. 

In analyzing the transformation agenda in South Africa, including within 
higher education, it can be conceived of as being riveted with paradoxes. If a 
paradox is understood as entailing an idea constituted by opposing 
propositions that, however contradictoiy they may be, are for good political 
and social reasons, equally essential for effective pursuit of the 
transformation agenda, creatively addressing the paradoxes is crucial to the 
effective functioning of higher education. 

A number of goals and/or strategies related to goals stand in a 
relationship of intractable tension and establish difficult political and social 
dilemmas and choices and decisions. The challenge is to find policies and 
strategies which, in the context of existing conditions, can satisfy multiple 
imperatives and can bolance and enable the pursuit of equally desirable 
go^s. This task represents a major and taxing challenge. 

For example: given the policy goals and challenges of both global 
competitiveness and redistributive reconstruction and development, a 
crucial question is posed for higher education. How does it orient itself 
towards both these imperatives? How are the varying needs of both these 
two poles to be satisfied simultaneously? More specifically, what does this 
challenge mean for individual higher education institutions or for groupings 
of higher education institutions - the historically advantaged and 
disadvantaged universities and technikons? Are all higher education 
institutions to be oriented towards both poles or is there to be a functional 
differentiation with respect to the two poles? Are these to be choices that are 
to be left to higher education institutions themselves or is the State to 
actively steer in this regard? 
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Notwithstanding the challenges, the existence of a vision and clearly 
defined purposes and explicit and concrete policy goals and objectives for 
higher education is a major strength of the South African transformation 
agenda. Further, since the formulation of explicit goals and objectives is a 
necessary condition for constructing strategic indicators, their existence 
facilitates the construction of system-level strategic indicators. 

3. SYSTEM-LEVEL INDICATORS FOR HIGHER EDUCATION: PAST, 
PRESENT, AND FUTURE 

3.1. The Past 

The previous system of higher education was driven by apartheid ideology 
and characterized by fragmentation along racial and ethnic lines. The role of 
the State was a mixture of political interference in some institutions (largely 
those historically black) and yet also the provision of a high level of 
autonomy for others (those historically white) institutions. Overall, there 
was little rationality and coherence in the system, an absence of overall 
national planning, and limited strategic institutional planning. No attention 
was given by government and higher education institutions to formulating a 
clear and participatory vision for the system arid to defining the purposes 
and goals of higher education and the pailicular missions of individual 
institutions. 

In the mid-1980s, the national Department of Education introduced an 
information system known as the South African Post-Secondary Education 
(SAPSE). Each higher education institution was required to submit 
information annually related to: 

- qualifications, fields, and levels of study; 

- students - headcount and full-time equivalents (FTEs) by race, sex, 
levels and fields of study, geographic origin, institutional origin, 
courses passed, diplomas/degrees earned, etc.; 

- staff - headcount and FTEs by race, sex, age, rank, qualification, and 
contact hours spent on teaching, research, and administration; 

- finances; 

- fixed assets; 

- buildings and space - lecture halls, laboratories, etc. 

This infoiTnation was considered necessary for “effective administration, 
planning, and policy formulation”. However, in the absence of clear, explicit 
policy goals and of a strong notion of public accountability, effectiveness 
was largely a technical matter, rather than related to the achievement of 
widely agreed upon national goals. In this context, there was, not 
surprisingly, no discourse about measuring and monitoring the 
performance of the higher education system and institutions, and there was 
no explicit reference to performance indicators or strategic indicators. 
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3.2. The Present and Future 

Since 1997, a new legislative and policy framework has emerged with an 
explicit vision for higher education and an explicit definition of its social 
purposes and of policy goals and objectives. Concomitantly, three 
mechanisms have been identified as key instruments for achieving policy 
goals and objectives. 

The first is iterative national and institutional planning processes, which 
have resulted in five-year institutional plans around teaching and research, 
three-year institutional “rolling plans”, and a National Plan for Higher 
Education (MoE, 2001). The second mechanism is a new framework for the 
funding of higher education, one which is very goal-directed (DoE, 2001). 
Finally, there is a new national quality assurance system centered around 
the compulsory accreditation of higher education learning programmes, 
programme reviews, and intermittent audits of institutions (HEQC, 2001). 
At the same time, there has been a convergence with international trends 
for greater public accountability, on the part of higher education 
institutions, for the expenditure of public funds and around their 
performance - the scrutiny of the efficiency, effectiveness, quality, 
responsiveness, and good governance of institutions. 

Thus, a discourse of measuring, monitoring, and evaluating the 
performance of higher education system/institutions has been emerging 
with explicit references to strategic and performance indicators. 

Within South African higher education there was an initiative to replace 
SAPSE, as of 2000, by a new Higher Education Management Information 
System (HEMIS). This system was developed and is managed by the 
national Department of Education, but with close attention paid to the easy 
incorporation of information generated by the quality assurance activities of 
the Council on Higher Education. 

HEMIS is intended to: 

- devote specific attention to data collection for measuring the 
performance of the system and institutions around key policy goals 
and objectives; 

- require each institution to submit information annually (even though 
this requirement is being limited with respect to certain data); 

- begin with the collection of student and staff statistical information ; 

- overcome the complexity and limitations of SAPSE; 

- continue to make use of SAPSE in other areas - for example, for the 
construction of performance indicators that can draw on information 
generated through the present SAPSE. 

4. SYSTEM-LEVEL INDICATORS: ISSUES AND PROBLEMS 

The identification of appropriate performance and strategic indicators is 
made easier by the clear identification of policy goals and objectives. 
However, national and institutional strategic planning is still an emerging 
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culture. The importance of performance and strategic indicators in this 
regard should not be considered as self-evident, but must be continuously- 
emphasized and demonstrated. 

The availability and use of strategic indicators depends, of course, on 
effective, efficient information systems. Despite some progress, the present 
higher education information systems continue to be sorely inadequate, 
especially in relation to information regarding financial matters. Moreover, 
many institutions lack the effective capacity to provide and to process data 
and information and have a very limited culture of reflective institutional 
research. Building a much more responsive and modem information system 
that provides policy-relevant real-time data on students and staff, both in 
the public and private higher education areas, is an urgent challenge. 

It will be necessary for the national Department of Education to develop 
the operating systems and capacity at national level, to also provide the 
overall framework for systems at institutional levels, and to help develop the 
operating systems and human capacity at Institutional levels. Ultimately, 
the co-operation of institutions in consistently submitting information that 
is accurate and timely is vital and must be secured. There is little reliable 
information presently on the operations and functioning of the private 
higher education sector, an issue that has to be addressed. 

The fields of policy monitoring and evaluation and strategic planning, 
within which performance and strategic indicators are embedded, are 
relatively new in South Africa. The capacities and expertise to generate 
indicators, to collect and to process information and data, and to analyze 
and interpret the indicators need to be developed at system and 
institutional levels. 

Finally, co-ordination and co-operation among different agencies and a 
confluence of activities related to information systems and the gathering of 
pertinent data will be imperative. The agencies in South Africa that are 
immediately linked to education and training include the national 
Department of Education, the Council on Higher Education, and the South 
African Qualifications Authority, which is responsible for the 
implementation of the National Qualifications Framework. Other agencies 
include government departments such as Labour (which is involved in high- 
level skill development at national level). Health (which regulates the 
nursing colleges), and Home Affairs (which issues visas for foreign students 
and staff). 

5. THE UNESCO PROJECT: CRITICAL ISSUES 

A commitment to the UNESCO World Declaration goals clearly necessitates 
a joint effort by countries to develop strategic indicators to measure 
progress and the achievement of higher education objectives. At the same 
time, it is important to address a number of issues of a conceptual, strategic, 
and empirical nature. 

To begin with conceptual issues, a number of crucial questions need to 
be addressed. First, it is important to achieve clarity around key concepts 
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such as “indicator”, “performance indicator”, and “strategic indicator” - both 
regarding what is distinct about each concept and also regarding the 
relations, if any, among them. Second, it is necessary to clarify the 
relationship between indicators and what is simply information /statistics, 
and between indicators and information systems. Third, the relation 
between “system-level” and “institution-level” indicators needs to be 
clarified? Are, for example, institution-level indicators the pre-condition for 
system-level indicators? Or are they not necessarily so tightly coupled, but 
just one input into system-level indicators? Fourth, what is the relationship 
between indicators and values and choices regarding the purposes, goals, 
and objectives of higher education? Fifth, are indicators inextricably linked 
to policy intentions and intended outcomes? Can there be indicators for 
capturing the unintended outcomes of policies? Sixth, is the focus to be on 
outcomes or structures, or mechanisms/instruments or processes, some of 
these, or all of these? Finally, there is the enormous challenge of making 
judgments on issues such as the character of “institutional culture”, the 
“quality of learning- teaching”, the “relevance” of institutions and 
programmes, student and staff “participation in governance”, and so forth. 

Turning to the strategic issues, one important task is to convince 
countries and important stakeholders of the importance and value of 
strategic indicators. If countries and institutions are not persuaded of the 
vital role that strategic indicators can play in developing more effective 
quality and efficient higher education provision, the necessary financial and 
human resources are not likely to be devoted to this activity. Another 
important task will be to ensure that countries do not experience the 
activity of generating system level strategic indicators as an external 
imposition, but have a genuine sense of ownership of the developments of 
processes and systems. Here, it is necessary to be sensitive to the highly 
uneven contexts of developed and developing countries and within 
developing countries with respect to higher education information systems. 
This understanding must influence choices as to how comprehensive or 
selective a range of strategic indicators is necessary, as well as the human 
and financial resources that are available to specific countries. Capacity 
development at institutional and individual levels may be a necessary 
condition for certain countries to be in a position to generate and provide 
UNESCO with the necessary strategic indicators. 

Finally, with regard to the empirical issues related to a project of 
developing strategic indicators, it is sufficient to highlight just two issues. 
First, it will be necessary to define clearly and explicitly what will be the 
products, outcomes, and processes related to the project. Second, it will 
also be essential to specify the time frames and timelines for the different 
phases of the project and the overall project. This task has a major bearing 
on many of the strategic issues that have already been raised. 
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6. CONCLUSION 

There is a high degree of congruence between the policy goals and actions 
that have been defined for South African higher education and those that 
are identified in the UNESCO documents emanating from the 1998 World 
Conference. For South Africa, with good progress around the new Higher 
Education Management Information System, producing the range of 
strategic indicators that may be agreed upon by UNESCO should be a 
relatively straightforward exercise which will not require any major 
additional resources. Indeed it could enrich its own efforts and even give 
impetus to its own commitment to generating strategic indicators for 
monitoring and enhancing the equity, quality, and the efficiency and 
effectiveness of its higher education system. 

However, as noted, it is necessaiy for UNESCO and its Member States to 
deal with a number of important issues, to arrive at workable solutions, and 
to communicate these effectively to countries. These issues are 
simultaneously conceptual, strategic, and empirical in nature, and their 
resolution will have a strong beai'ing on the success of any UNESCO 
initiative around system-level strategic indicators. 
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XI. Key Issues in the Development of 
Higher/Tertiary Education in China 

HONG-WEIMENG 

1. INTRODUCTION 

The remarkably high and sustained levels of economic growth in China are 
placing increasingly heavy demands on the skills and knowledge of its 
population, particularly of its senior scientists, technicians, and 
administrators. This situation has raised the need for the fundamental 
renewal and re-orientation of higher education at system and institutional 
levels in order [for it] to adjust to the demands of social-economic 
development. More importantly, a strategic shift has occurred in Chinese 
higher education from an elite-oriented to a public-oriented system. 

The Educational Statistics Yearbook of China (1990-1999) has proven 
to be a valid and valuable data collection organ for the monitoring and 
the evaluation of development and progress in higher/tertiaiy education, 
as well as in the whole education system. In order to be able to reinforce 
policy and decision-making, the annual data collection system should be 
further strengthened so as to be able to reflect key policy-relevant issues, 
and more particularly, to undertake factual reporting. 

This study will present a brief description of the higher/tertiaiy 
education system in China based on ISCED 1997 and highlight some key 
issues in its development. It may form the basis for further discussions on 
the development of relevant international comparative indicators. 

2. THE CURRENT STATUS OF THE HIGHER/TERTIARY EDUCATION 
SYSTEM IN CHINA 



Table 1 . Categories of tertiary education 


in China (based on ISCED 1997) 


Tlieoretical duration 


Level 5 - First stage of tertiary education 
(not leadmg directly to an advanced research qualification) 


Level 6 - Second 
stage of tertiary 
education 
(advanced research 
qualification) 




5B 


5 A 






First Second 

qualification qualification 


T- j ^ Second Research 

First degree 




2 and less thcui 3 years 

3 and less than 4 years 

4 and less than 5 years 

5 cind less than 6 years 

6 to 7 years 
Over 7 years 


Diploma 

Diploma 

Diploma 

Diploma 


Bachelor’s 

Bachelor’s Bachelor’s 

Research Master’s 
Programme Degree 


Doctoral Degree 



Source: The author. 
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3. THE ISSUES OF HIGHER/TEFTTIARY EDUCATION 

Higher/tertiaiy education in China developed rapidly in the last decade of 
the Twentieth Centuiy, but it is still relatively less developed than the 
average level of the lower middle-income countries around the world, in 
terms of gross enrollment rate. 

The gross enrollment rate is one of the key indicators for the 
development of higher/tertiaiy education in a country. In terms of this rate, 
higher/tertiaiy education in China expanded from 3.7 percent in 1990 to 
10.5 percent in 1999 and 11 percent in 2000. Figure 1 (below) illustrates 
the growth of the gross enrollment rate in higher/tertiaiy education in 
China from 1990 to 1999. 



Figure 1 . The growth of the gross enrollment rate of higher/tertiaiy education in 
China in the 1990s 
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Source: Ministry of Education, People’s Republic of China (1990-1999). 



According to the World Bank Report, Higher Education in Developing 
Countries (World Bank, 2000), the gross enrollment rate of higher/ tertiary 
education in China in 1999 is at average level for low- and middle-income 
countries around the world. Figure 2 shows the comparison of the gross 
enrollment rate of the higher/tertiaiy education in China, in 1999, with 
low- and middle-income countries, high-income countries, and the 
average world level. 
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Figure 2. International comparison of the gross enrollment rate for tertiary 
education 




Group of Countries 



Source: The World Bank (2000). 

Figure 3 presents a eomparison of the gross enrollment rate for 
higher /tertiary edueation in China with that of eertain developed and 
developing eountries. It again illustrates the reality that the gross 
enrollment rate the higher /tertiary edueation in China is relatively low. The 
results have had a signifieant impaet on poliey makers. An important 
deeision taken by the Chinese government in 1999 was “to expand the 
[numbers of] new entrants into higher/tertiary edueation” with a view to 
raising the edueational level of the population and to implementing a 
strategy to rejuvenate the eountry through seienee and edueation. 
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Figure 3. Comparisons of the gross enrollment rate into tertiary education of selected 
countries 




u u 



Source: The World Bank (2000). 



Countries 



In 1999, the “Decision of the CPC Central Committee and the State 
Council on the Development of Educational Reform and the Promotion of 
Quality-Oriented Education in an All-Around Way” and the “Action Scheme 
for Invigorating Education towards the Twenty-First Century” of the 
Ministry of Education were formulated. These documents set the goals and 
the orientation for the development of all types of education, at all levels, 
and for macro-stmctural readjustments through 2010. 

Figure 4 illustrates the rise in the numbers of new entrants into tertiary 
education during 1996-1999. 

To increase the opportunities for higher /tertiary education for young 
people and all citizens will be one of the major targets of national 
educational development over the decade. The national goal is to achieve a 
15 percent gross enrollment rate in higher/ tertiary education by 2010. 

The Government is tapping new resources and increasing the offerings 
for higher/ tertiary education. 
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Figure 4. The increase in the numbers of new entrants into tertiary education in 
China during 1996-1999 
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Source: Ministry of Education. People’s Republic of China (1996-1999). 

In order to achieve this goal, the government will encourage and support 
all sectors of society to run schools in various forms so as to meet the 
increasingly popular demand for education and to create a pattern whereby 
the government assumes the main responsibility for running schools, with 
public and private schools developing side by side. The government has also 
suggested that the State accelerate legislation regarding private education 
and promote its sound development. This policy decision made is very 
important. It has a direct bearing on the development of higher/tertiaiy 
education and the overall situation of education in the country. 

Table 2 presents the numbers of private higher/ tertiary education 
institutions. The numbers of students enrolled in these institutions 
increased significantly from 1986 to 1999. 



Table 2. The increase in tire numbers of private higher/ tertiary education 
institutions in China 





1,986 


1,991 


1,994 


1,995 


1,996 


1,997 


1,999 


Numbers of private 
higher/ tertiary education 


370 


450 


880 


1,209 


1,219 


1,115 


1,277 


institutions 

Numbers of students enrolled 
(10,000) 




None 




1 14.5 


120.4 


123 



Source: The author. 
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In 1999, thirty-seven private institutions were approved by the Ministry 
and qualified to award higher education certificates. These certificates are . 
recognized by the State. Most of the schools offer two or three years of 
professional training or vocational and technical education. Only one of 
them offers four years of education. 

Figure 5. Increased enrollments in private higher/tertiary education institutions 




Source: National Center for Education Development Research (2000). 

The second category is composed of the pilot schools set up with a view 
to preparing students for the higher education certificate examination. The 
students enrolled in such schools can obtain a state-recognized higher 
education certificate after passing the specially designed unified higher 
education certificate examination. 

The schools in the third category do not lead to state-recognized 
certificates. Most of them are guidance institutions for assisting self- 
learning students in passing examinations. These institutions give on-site 
instruction, offer correspondence courses, and help students participate in 
the yearly independent-study higher education examination. 

Table 3 indicates the varied categories of private higher/tertiary 
education institutions. 



Table 3. Private higher/tertiary education institutions, in China (in whole numbers). 





Institutions Students 


Institutions Students 


Institutions Students 




1996 






1997 




1999 


1. Qualified to issue 
certificates 


21 


1.4 


20 


1.4 


37 


4.6 


2. Experiments with 
examinations 


89 


5.1 


157 


9.4 


370 


25.8 


3. Others 


1109 


108 


938 


109.6 


870 


92.6 



Source: The author. 
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Private higher/ tertiary education in China meets market demands, offers 
distinctive features, and seeks to exist and to develop. It is gradually 
becoming an important part of the developing Chinese higher/ tertiary 
education sector. 

Recently, the jointly run Chinese and foreign higher/ tertiary institutions 
have attracted many Chinese students who intend to study abroad. These 
institutions have been legally established and may award certificates. This 
innovation has produced notable results in various areas. 

4. RESPONSES TO SOCIAL DEMAND 

It is obvious that the development of higher/ tertiary education is closely 
linked to the demands of socio-economic development. The information age 
is having its effect upon the social and economic structure of China. The 
rapid development of the information and communication technologies is 
making heavy demands on the human resources in the field. Whether or 
not higher education can offer a timely response to social demand will 
interest policy makers. The analysis of the most popular study fields in 
universities and colleges, in the last five years, offers assurance that the 
higher/ tertiary education institutions of China have the ability to satisfy 
social demand. 



Table 4. List of the twenty most popular study fields in 1999. 



Rank 


Field of study 


New entrants 


Total new entrants 
(percent) 


01 


Computer Science and Technology 


110.334 


7.12 


02 


Chinese Language and Literature 


72.251 


4.67 


03 


Machines and Automation 


71,453 


4.61 


04 


English Language and Literature 


69.068 


4.46 


05 


Clinical Medicine 


58.460 


3.78 


06 


Business Management 


57,134 


3.69 


07 


Accounting 


54,850 


3.54 


08 


Mathematics and Applied Mathematics 


47,834 


3.09 


09 


Law 


42,982 


2.78 


10 


Information Engineering 


37,820 


2.44 


11 


Electrical Engineering and Automation 


35,296 


2.28 


12 


Electronics Science and Technology 


34,758 


2.24 


13 


Economics 


33,059 


2.13 


14 


Automation 


30,193 


1.95 


15 


Architecture 


27,231 


1.76 


16 


Physics 


26,110 


1.69 


17 


Physical Education 


24,304 


1.57 


18 


Chemistry 


23,811 


1.54 


19 


Marketing 


23,412 


1.51 


20 


Civil Engineering 


22,906 


1.48 


Soujce: The author. With the numbers of new entrants enrolled in higher/tertiary 
mind. Table 4 lists the first twenty most popular study fields in 1999. 


education across China i 
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The results show that: 

- Among the twenty most popular study fields, “Computer Science and • 
Technology” is the most significant one, as the numbers of new 
entrants in it were the highest for new entrants in 1999. 

- Four of the twenty most popular study fields are related to computer 
and information technology. The total numbers of new entrants in 
these study fields amounted to 13.77 percent of the total numbers of 
new entrants across the country. 

Table 5 indicates the study fields having enrolled the greatest numbers of 
students over the 1995-1999 period. 

Table 5. Most popular study fields in Chinese higher/ tertiary higher education: 
1995-1999 



Rank Study field 


New entrants (in whole numbers) 


01 


Computer Science and Technology 


326,941 


02 


Machines and Automation 


279,017 


03 


Chinese Language and Literature 


267,133 


04 


English Language and Literature 


242,018 


05 


Accounting 


222,898 


06 


Clinical Medicine 


22,2230 


07 


Business Management 


201,335 


08 


Mathematics and Applied Mathematics 


181,610 


09 


Law 


138,175 


10 


Economics 


124,814 


11 


Electrical Engineering and Automation 


119,455 


12 


Information Engineering 


116,715 


13 


Automation 


110,127 


14 


Physics 


106,942 


15 


Civil Engineering 


105,919 


16 


Physical Education 


98,930 


17 


Chemistry 


98,573 


18 


Chemical Engineering and Technology 


96,656 


19 


Electronics Science and Technologies 


89,101 


20 


International Trade and Economics 


83,630 



Source: The author. 



5. STRENGTHENING THE POSITION OF HIGHER/TERTIARY EDUCATION 
IN THE NATIONAL KNOWLEDGE INNOVATION SYSTEM 

In 1998, the Chinese government began to elaborate a national knowledge 
innovation system and supported the establishment of top class universities 
and study fields by means of which China might catch up with the most 
advanced nations of the world. 

The Ministry of Education initiated its “Project on Senior Innovative 
Personnel in Higher/Tertiary Education Institutions” and “Project on High- 
Tec Industrialization Aided by Higher /Tertiary Education Institutions”. The 
projects have been designed to encourage higher /tertiary education 
institutions to focus on key practical problems arising in national economic 
and social development, to take full advantage of innovative thinking and 
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personnel, and to organize personnel in study fields with notable problem- 
solving capacities in science and technology. In the meantime, increasing 
numbers of specialists and scholars in higher/tertiaiy education 
institutions are involved in consulting and research and in surveys and 
analyses of macro-policies at national and regional level. 

At present, two-thirds of the national key laboratories are located in 
universities. Some provincial and ministerial laboratories have also been set 
up in universities and colleges. All these constitute an important part of the 
national knowledge innovation system. 

Over the past decade, the varied forms of co-operation between the 
higher/tertiaiy education institutions, research agencies, and industrial 
enterprises were established and played an increasingly significant role in 
regional economic and social development. 

University science parks represent a new form of such co-operation. They 
have become bases for incubating innovative technical enterprises, for 
training innovative entrepreneurs, and for developing high-tech industiy, 
markets for scientific and technological achievements, and centers for 
financial information. In 2000, the Ministiy of Education and the Ministiy of 
Science and Technology jointly designated fifteen national science parks to 
be attached to universities on an experimental basis. 

More such arrangements will be undertaken in the coming decades. The 
construction of science parks in universities has been incorporated into the 
national ‘Torch Programme” for tackling hard-nut problems in science and 
technology. From 2001 to 2005, the government will establish some 100 
national science parks to be gradually attached to universities. 

6. THE EXPANDING ROLE OF THE INFORMATION AND COMMUNICATION 
TECHNOLOGIES AND THE INTERNET IN THE DELIVERY OF HIGHER/ 
TERTIARY EDUCATION AND IN CREATING A LIFELONG LEARNING 
SYSTEM 

Owing to limited resources in the development of higher/tertiaiy education, 
the distance higher/tertiaiy education network has been highly valued by 
the population. The high enrollment figures in the China TV/Radio 
University are good evidence in this respect. 

The distance education system for higher/tertiaiy education was 
established through the use of satellite television which has remained the 
principal media since the late 1970s. Recently, the development of 
information and communication technologies has brought about the 
possibility of using the information and communication technologies (ICTs) 
and the Internet in the field. At present, the China Education and Research 
network (CERNETT) is the largest academic network in China with over 2 
million users. Since 1999, the central government has been allocating 
special funds to build a high-speed main network. All the universities and 
colleges are linked to CERNETT and to the Internet. 

In 1999, six universities - among them the University of Beijing and 
Tsinghua University - opened network universities (e-universities). By the 
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2000-2001 academic year, the Ministry of Education had approved thirty- 
one network universities. They will help train senior special personnel as 
public servants. More importantly, the network universities can ease the 
effects of contradictions caused by the irrational distribution of resources 
for tertiary education. Increasing numbers of people wlU have the 
opportunity to receive quality higher/ tertiary education through the 
network. And it will provide employees and citizens with a more effective 
means to continue their lifelong learning endeavours. 
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1. INTRODUCTION 

This study presents a Japanese perspective on the current status and 
problems related to the design and use of system-level indicators for 
higher/tertiary education. 

First, the historical development of system level information on Japanese 
higher/tertiary education will be explained, especially in relation to 
Japanese trends in higher/tertiary education policies. Second, this study 
examines the current status of higher/tertiary education in Japan in 
relation to the UNESCO Action Plan. Third, it considers the possible further 
development of Japanese indicators in the context of the development of 
system-level strategic indicators by UNESCO. 

2. DEVELOPMENT OF SYSTEM-LEVEL INDICATORS 

2. 1 . Historical Development of Statistics on Higher /Tertiary Education in Japan 

Statistical data collection on higher/ tertiary education has a long history in 
Japan. The Ministry of Education (MEXT) has been collecting basic data on 
higher/tertiary education since the establishment of modem public 
higher/tertiary education in Japan in the latter half of the Nineteenth 
Century. The Ministry of Education has been publishing its Annual Report 
since 1875. This Annual Report has been reporting basic information 
concerning higher/tertiary education - numbers of schools, students, and 
staff - in a frequently changing content and format. 

After the Second World War, the basic stmcture of the curr'ent 
higher/tertiary education system was devised. To follow twelve -year primary 
and secondary education, four-year undergraduate programmes and two- 
year' Junior college programmes were introduced, according to the American 
model. Four-year undergraduate programmes were introduced in 1949, and 
junior college programmes, in 1950. In addition to the university and junior 
college systems, the system of colleges of technology was established in 
1962, as one of the types of higher/tertiary education that provides five-year 
education following nine years of compulsory education. In 1976, the 
specialized training college system started as two-year programmes of non- 
university higher/tertiary education. 
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In 1948, the Ministry of Education began compiling the School Basic 
Survey that has been published since 1952 as an independent publication.- 
This survey lists the numbers of schools, students, applicants, and 
enrollments, as well as the numbers of graduates and teaching and non- 
teaching staff members. It also gives information on the destination of 
students after graduation, on graduate schools, and on the financing of 
public institutions (MEXT, 1948-2000). 

The Ministry of Education has also endeavoured to collect infoirnation 
concerning overseas higher /tertiary education. It has a very strong division 
for the collection of data and information concerning foreign education. In 
general, this approach has worked quite well. It has been able to provide the 
Japanese government with data that are not available in the existing data 
sets collected and reported by international organizations. 

The Division of Survey and Statistics in the Ministry of Education has 
been publishing its annual report. International Comparison of Educational 
Indicators, since 1969 (MEXT, 1969-2000). The successive issues of this 
report publish basic indicators such as enrollments, participation rates, the 
proportion of graduate students, student/teacher ratios, educational 
expenditures, tuition fees, and the student loan schemes of “competitors”, 
Le., the United States, the United Kingdom, France, Germany, Russia, and 
China. Moreover, the division has published many survey reports and data 
on educational systems all over the world. 

Comparative studies by academics have also contributed to the 
development of indicators and information concerning higher/ tertiary 
education. In 1989, the Research Institute for Higher Education of 
Hiroshima University published the first longitudinal data collection on 
higher/tertiary education in Japan (Research Institute for Higher 
Education, Hiroshima University, 1989 and 1995). 

Other related data, such as data on labour, population, industry, science 
and technology, and family income have also been collected by various 
ministries and institutions. 

2.2. Introduction of Planning in Higher /Tertiary Education 

Higher/tertiary education policies have relied heavily, and for a long time, 
on the qualitative and quantitative analyses of collected data. In this 
respect, the Japanese government has made use of many indicators for 
decision-making in the elaboration of higher/tertiary education policies, 
sometimes quite strategically. However, it is also true that other social and 
political factors have influenced higher/tertiary education policies. 

Before the Second World War, public higher/tertiary education 
institutions were established on the basis of geographically balanced public 
planning. However, the rapid expansion of the private sector in the 1960s 
made it almost impossible for the national government to limit the over- 
concentration of institutions in the larger cities. Even in 1951, about 70 
percent of university students lived in the major urban areas, and more 
than 60 percent of private university students lived in Tokyo. 



THE JAPANESE PERSPECTIVE . 161 



In the late 1960s, a serious attempt was made to prepare policy reports 
with comprehensive data analyses. The 1971 report by the Central 
Education Council consisted of comprehensive analyses of the development 
of the modem education system and projections regarding future 
enrollments in higher/tertiaiy education (Central Education Council, 1971). 
The report was used as a powerful policy document to support the 
introduction of the Higher Education Plan in 1976 (Kuroha, 2001). 

At that time, the balance between the quality and the quantity of 
higher/tertiaiy education became a most critical issue. Japan experienced a 
rapid and unprecedented expansion of higher/tertiaiy education in the 
1960s. This expansion led to the weakening of the quality of education and 
to political controversy, including the stimulation of student movements. 

Another important issue, at that time, was that of the equality of access 
for different economic and social groups. In Japan, the mass higher/tertiaiy 
education system has relied heavily on the private sector. The difference 
between the sizes of the tuition fees charged, respectively, by the public and 
private sectors became a cmcial inhibitor of equality of access. This 
situation led to the introduction of public subsidies for the operational costs 
of private institutions. 

The over-concentration of higher/tertiaiy institutions in the big cities 
also became a political issue. The rapid expansion of mega-cities, such as 
Tokyo and Osaka, in the process of economic “take-off’, attracted young 
people to live in them. As private institutions placed their campuses so as to 
maximize their economic benefits, the drain of young people from local 
areas into city areas became a critical issue. The government introduced the 
Higher Education Plan in 1976, and the establishment and expansion of 
schools in large cities were prohibited. 

Following the introduction of the Plan, the quality of higher/tertiaiy 
education improved a great deal, especially in the private sector. The 
quantitative indicators revealed improved “quantitative aspects” in the 
quality of education. 

The accessibility of women as students to four-year higher/tertiaiy 
education programmes still remains an important challenge. Women still 
face difficulties in obtaining emplojmient after graduation; however, a law 
calling for equality in recruitment was enacted in 1986. 

2.3. Quantitative Indicators to Qualitative Indicators 

In the 1990s, the focus of university reform moved to the qualitative aspects 
of education. Organizational reform, the introduction of self-monitoring and 
evaluation, and curriculum reform were monitored through national 
surveys by academics, ministries, and other associations and companies 
(MEXT, 1999). 

Commercial university guides also established an information market on 
higher/tertiaiy education institutions, including multiple indicators 
regarding student satisfaction, academic prestige, and other kinds of 



162 



A.ARIMOTO, A. YONEZAWA, H. FUKUDOME, and M. TAKEUCHl 



indicators. At least to some degree, these developments have contributed to 
university reform. 

2.4. Enter the Institutional Level Data Base System 

The most recent transformation in Japan is that the various institutional 
data have become available, adding to the traditional system-level aggregate 
data. Each institution, particularly since the 1990s, has been asked to cany 
out self-evaluation and to periodically publish these data as self-study 
reports. The total number of these reports is already legion. 

In addition, several national institutes for specific tasks have provided 
databases concerning institutional-level data. The National Center for 
University Entrance Examinations (NCUEE) founded an online information 
system, called the Heart System, in order to provide applicants with 
information on universities. In 2001, the Heart System became an Internet- 
based service. Both quantitative and qualitative information can be drawn 
from this online database. 

The National Institute of Informatics (Nil) provides a database on 
individual researchers, academic work, public databases, etc. This 
information is quite important to the development of qualitative indicators 
related to research and development. 

The National Institution for Academic Degrees (NIAD) has started a pilot 
project on quality assessment for national universities. Assessment will be 
implemented for education and research activities according to acadernic 
fields. At the same time, NIAD is implementing a thematic review. In this 
activity, all the national universities will be assessed both at system-level 
and at institution-level according to specific criteria. 

3. THE CURRENT STATUS OF HIGHER/TERTIARY EDUCATION IN JAPAN 
- WHAT THE INDICATORS REVEAL 

3. 1 . Can One Say that What the Indicators Reveal Has Already Been Achieved? 

The Japanese government has accomplished most of the tasks set out in 
the two UNESCO policy documents: Higher Education in the Twenty-First 
Century (UNESCO, 1998a), and Framework for Priority Action for Change 
and Development of Higher Education (UNESCO, 1998b), through reference 
to indicators available from public documents. However, it is obvious that 
most of these indicators need to be viewed as “qualitative”, and that some 
goals are sometimes too abstract to be identified by appropriate quantitative 
indicators. 

3.2. Equity of Access 

Japan had achieved mass higher/ tertiary education by the early 1970s. The 
decrease in the number of secondary school graduates and the continued 
establishment of new higher/tertiary education institutions contributed to 
the present situation in which 49.4 percent of high school graduates 
continue to study in four-year universities and junior colleges and 70.5 
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percent in higher/tertiary education institutions as a whole (Figure 1). 
According to a simulation exercise of the University Council, by 2008, all 
eighteen-year old students who wish to enter four-year universities and 
junior colleges will be admitted somewhere in an institution of this sort, by 
the effect of the decline in the numbers of the eighteen-year old cohort. As a 
result, the access ratio to colleges and universities is expected to increase 
up to 60 percent of the cohort. 

Figure 1. Percentage of eighteen-year olds studying in various Japanese higher 
education institutions (1955-2000) 




1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 

Academic Year 



♦ 4 -year Universities (Undergraduate) 

— □ — Junior Colleges 
— ^ — Colleges of Technology 

—Specialized Training Colleges 

— ^ — Higher Education as a Whole 

Source: MEXT, Gaicko kOion Chosa hokolaisho (Report of school basic survey) (1955-2000), 



3.3. Enhancing the Participation and the Promotion of Women 

At present, the total number of women enrolled in higher/tertiary education 
is almost the same as that of men. However, there are large differences in 
the proportions of women among different types of higher/tertiary 
education. The percentage of women studying in the university sector 
(undergraduate) is relatively low. Meanwhile, almost 90 percent of the junior 
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college students are women. Also, over half of the students in specialized 
training colleges are women. In short, educational opportunities for women 
are limited to those providing higher/tertiary education over relatively short 
study periods (Figure 2). 

Figure 2. Percentages of women students in various stages of higher education 




■■ ♦ Higher Education as a Whole 
— (33 — Universities (Undergraduate) 

— ^ — Junior Colleges 

— H — Colleges of Technology 

— ^ — Specialized Training Colleges 

Source: MEXT, Gakko kihon Chosa hokokusho (Report of school basic survey) (1955-2000). 
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3.4. Advancing Knowledge through Research in Science, the Arts, and the 
Humanities, and the Dissemination of Results 

The budget for Research and Development (R&D) increased through the 
1990s. At the same time, the linkage between the universities and industry 
has recently been strengthened (Arimoto, 2000; Asonuma, 1999). The 
implementation of science policy was affected by the integration, in 2001, 
of the former Ministiy of Education and the former Ministry of Science and 
Technology. The Ministry of Education published Indicators for Science and 
Technology 2000 (MEXT, 2000). The numbers of graduate students and 
researchers, along with indicators showing outcomes, such as the numbers 
of publications and citations, are included in this report. According to these 
data, the number of publications by Japanese researchers is the second 
largest among the seven major industrial countries (the proportion is over 
10 percent following about 33 percent for the United States, which is 
exceedingly high), but the index of the relative citation of these publications 
is the lowest among the countries in question. 

3.5. Strengthening Co-operation with the World of Work and Analyzing the 
Anticipation of Societal Needs 

Education and training, with a view to improving problem-solving abilities, 
are being stressed in current higher/tertiary educational reforms. The 
linkage between the world of work and university education, particularly at 
graduate level, is becoming important, and increasing numbers of workers, 
especially the people employed in professional work, are returning to 
campus to obtain systematic skills. 

The lifelong emplo 3 nnent tradition in Japan is being questioned. The 
boundary between education and work is becoming unclear. Many 
universities are allowing adult students to enter by special admission, and 
more and more adult students are making use of this system. However, the 
Japanese public survey lacks precise data on student numbers by age 
group. Data on lifelong learning should be developed to enable an 
understanding of current changes in the labour customs in Japan. 

3.6. Diversification for Enhanced Equity and Opportunity 

The Japanese higher /tertiary education system is highly diversified. Private 
higher/tertiary education is helping to sustain mass higher/tertiary 
education: 76 percent of the institutions are private, and 78 percent of the 
total number of students are enrolled in these institutions (Table 1 and 
Table 2). 

Also, the diversity of educational programmes is increasing. The 
indicators that can be utilized to measure this change include the 
following: deregulation of admission requirements and of the number of 
years needed to graduate, the increase of non-degree students and transfer 
students, special admission for adult students, development of the credit 
transfer system, evening graduate programmes, etc. 
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Table 1. Number of higher education institutions in Japan by type and control (2000) 





Total 

(Universities, 
Junior Colleges, 
Colleges of 
Technology) 


Universities 


{Universities 
which provide 
graduate 
education) 


Junior 

Colleges 


Colleges of 
Technology 


Specialized 

Training 

Colleges 


Total 


1,283 


649 


479 


572 


62 


3,O03 


(%) 


100.0 


100.0 


100.0 


100.0 


100.0 


100.0 


National 


173 


99 


99 


20 


54 


130 


(%) 


13.5 


15.3 


20.7 


3.5 


87.1 


4.3 


Local public 


132 


72 


50 


55 


5 


208 


(%) 


10.3 


11.1 


10.4 


9.6 


8.1 


6.9 


Private 


978 


478 


330 


497 


3 


2,665 


(%) 


76.2 


73.7 


68.9 


86.9 


4.8 


88.7 




In addition, there is one Air University. 







Source: MEXT, Gakko kihon chosa hokokusho (Report of school basic survey) (2000). 



Table 2. Numbers of students in higher education institutions by type and control 
( 2000 ) 





Total 

(graduates of and 
undergraduates 
in junior colleges 
and colleges of 
technology) 


Graduates 


Undergraduates 


Junior 

colleges 


Colleges of 
technology 


Specialized 

training 

colleges 


Total 


3,663,060 


205,311 


2,471,755 


327,680 


21,006 


637,308 


{%) 


100.0 


100.0 


100.0 


100.0 


100.0 


100.0 


National 


641,316 


128,624 


471,631 


7,772 


18,417 


14,872 


(%) 


17.5 


62.6 


19.1 


2.4 


87.7 


2.3 


Local public 


156,820 


9,719 


9,3062 


21,061 


1,652 


31,326 


(%) 


4.3 


4.7 


3.8 


6.4 


7.9 


4.9 


Private 


2,864,924 


66,968 


1,907,062 


298,847 


937 


591,110 


{%) 


78.2 


32.6 


77.2 


91.2 


4.5 


92.8 



Source: MEXT, Cakko kihon chosa hokokusho (Report of school basic survey) (2000). 



3. 7. Innovative Educational Approaches 

In general, the Japanese Government is recommending innovative and 
unique educational approaches. Recent educational approaches in 
Japanese higher/tertiary education that can be indicated by specific data 
include the following: the reform of education for information processing, 
the introduction of new courses regarding volunteer activities, the adoption 
of the semester system, the dissemination of systematically compiled syllabi 
among students, the equipping of the distance education system, the use of 
teaching assistants, the deregulation of admission requirements and of 
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study years required for graduation, the reform of language education, the 
increase in the number of small classes. 

3.8. Higher /Tertiary Education Staff and Students as Major Actors 

National policy strongly recommends the setting up of faculty and staff 
development programmes. Thus, many universities are adopting them. 
According to the international study on academic professions of the 
Carnegie Foundation for the Advancement of Teaching (Arimoto, 1996), 
Japanese professors tend to categorize themselves as researchers rather 
than as teachers in terms of the priorities they accord to involvement in 
academic work. Among the respondents to the relevant questionnaires, 
approximately 70 percent of Japanese academics call themselves 
researchers, and 30 percent stress that they are teachers. 

This traditional view that probably stemmed from the German model is 
running into difficulties at a time when the improvement of teaching is 
increasingly required in order to respond to the massification of 
higher /tertiary education. In general, ^most all faculty in Japanese 
universities were hired, until quite recently, with tenure. Based upon a new 
law concerning faculty emplo 5 nnent, enacted in 1997, the introduction of a 
non-tenure employment system is being recommended. 

As an example of the indicator of this policy change, a government 
survey indicates that, recently, twenty-one universities have adopted non- 
tenure systems for some of their faculty positions. In 1999, there were 
ninety-nine non-tenured posts in total. At the same time, the practice of 
student evaluation of educational programmes in Japan is rapidly 
expanding. 

3.9. Qualitative Evaluation 

In the 1990s, a self-evaluation system was circulated among Japanese 
higher/tertiary education institutions. As of 1999, 533 universities or 88 
percent of the total number of institutions were appl)dng self-evaluation. In 
addition, 94 universities (16 percent) invited external examiners to verily 
their self-evaluations. NIAD has established a system for the evaluation of 
the national universities. Also, the Japanese University Accreditation 
Association (JUAA) has been working for about sixty years, since its 
establishment in 1946. Accreditation boards for professional or specialized 
study areas, such as the Japan Association Board of Engineering Education 
(JABEE) began to engage in accreditation activities aimed at obtaining 
international recognition for their qualifications. Related to qualitative 
evaluation, arguments inside and outside academe are being voiced about 
the scholastic abilities of students. Nowadays, universities should cater to 
student diversity at entry, before they begin studying at higher /tertiary 
level. Adequate and accurate statistics on student preparedness based on 
nationwide surveys of student abilities are needed. There is also a need to 
establish standards of student scholastic ability at higher/tertiary 
education level. 
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3.10. Strengthening Higher /Tertiary Education Management and Financing 

Up until now, all of the national universities in Japan have been regarded 
as a part of the system administered by the Ministry of Education. However, 
their status as national institutions is now in the process of transformation 
so that each one will have the status of an “Independent Administrative 
Corporation”. The introduction of corporate-style management into 
universities requires the strong leadership of the presidents of each 
institution. However, there is no proper indicator for this type of 
empowerment for university heads. 

3.11. The Financing of Higher /Tertiary Education as a Public Service 

There are four streams of major government allocations for higher /tertiary 
education: (i) funds transfers from a special account to the national schools; 
(ii) current cost subsidies to private higher/tertiaiy education institutions: 
(Hi) science research grants; (iv) credit loans to the Japan Scholarship 
Foundation. In 1998, each item occupied 74.7 percent, 14.4 percent, 5.7 
percent, and 5.2 percent, respectively, in the total government expenditure. 



Table 3. Government expenditure for higher education by country (as a percentage of 
GDP), 1997 



Percenta^^e of GDP 


The United States 


1.4 


The United Kingdom 


0.7 


France 


1.0 


Germany 


1.0 


Canada 


1.2 


Australia 


1.0 


Denmark 


1.1 


Italy 


0.6 


The Netherlands 


1.1 


Spain 


0.9 


Austria 


1.3 


Sweden 


1.6 


Switzerland 


1.1 


Japan 


0.5 



Source: Daigaku Singikai (University Council). Global-ka Jidai ni motomerareni koto kyoika no ariicata ni tsuile 
(A Vision for Higher Education in the Global Society) (2000), p. 90. 



The national government started to subsidize private institutions in 
1970, based on the concept that equal opportunity for education should be 
protected by the national government. As the higher/tertiaiy education 
system is heavily dependent on the private sector, the amount of public aid 
to private institutions could be a good indicator. Also, recently, the numbers 
of student loans offered by the Japan Scholarship Foundation have rapidly 
increased. 

In Japan, the rate of government allocations from GDP for 
higher/tertiaiy education is seriously low when compared with the rate in 
other major developed countries. For example, in 1997 the government of 
the United States spent a total of 1.4 percent of its GDP on higher/tertiaiy 
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education, while that of Japan, only 0.5 percent (Table 3). However, as a 
result of the current recession, it is difficult for the Japanese government to 
improve this situation, therefore each institution is being forced to secure 
its own capital. 

3.12. Globalization 

The number of students in Japan from foreign countries is increasing. Most 
of them are coming from Asian countries, particularly from China and 
Korea. In 1999, students from these two countries represented 46.5 percent 
and 21.3 percent, respectively, of the total percentage of foreign students. 

The problem is that only small numbers of students come from countries 
other than Asian countries. For example, only 1.9 percent of all foreign 
students are from the United States, and this rate is the highest for all the 
developed countries. Another problem is that only a small proportion of 
foreign students come to Japan with grants or scholarships from their home 
countries. In 1999, only 18.5 percent of them received such grants or 
scholarships. 

The numbers of faculty members from foreign countries are rapidly 
increasing. In 1998, there were 4,612 full-time faculty members from 
foreign countries in Japanese four-year universities. These numbers more 
than doubled through the 1990s: in 1990, there were only 2,183 foreign 
university professors. 

Japanese students who study abroad have always exceeded the numbers 
of foreign students stud 3 dng in Japan. In 1999, more than 75,000 Japanese 
students were stud 3 dng abroad compared with 55,755 students from foreign 
countries stud 3 dng in Japan. 

In 2000, the University Council published its report on the strategic 
planning of higher /tertiary education in a global society. The rapid 
development of e-leaming among Asian countries is exerting heavy pressure 
for the internationalization of Japanese higher/ tertiary education. 
Indicators for this phenomenon need to be developed. 

3.13. Partnerships and Alliances 

There are several local, national, and international partnership bodies and 
alliances involving Japanese higher/tertiary education institutions. A major 
example is the alliance linking four national universities in the Tokyo area: 
the Tokyo Medical and Dental University, the Tokyo University of Foreign 
Studies, the Tokyo Institute of Technology, and Hitotsubashi University. 
These four universities co-operate with one another for the exchange of 
course credits and in other ways. 

Another example is the Kyoto Consortium. Fifty universities in Kyoto, 
including national, local public, and private institutions, co-operate through 
this organization for credit transfer, internships, public lectures, 
information exchanges, and the effective use of educational resources for 
students and society. The mutual recognition of the credit system could be 
an indicator that shows the dissemination of this kind of partnership and 
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alliance. Partnerships and alliances are thought to be useful, and more are 
being established. However, quantitative indicators of these developments • 
are not available. 

4. FURTHER DEVELOPMENTS AND CHALLENGES 

The comprehensive domestic data collection system provides useful data, 
well suited for the demands related to the specific social context. However, 
for the further development of indicators and information on higher/ tertiary 
education in Japan, some aspects of the present data need to be expanded. 
For example, in the field of lifelong learning, the data are not well developed 
because of the unique structure of the labour market. In addition, most of 
these data are not comparable with international common data sets, for 
example, those of OECD. 

Current trends in globalization increase the necessity for international 
discussions and/or joint networks to formulate international indicators. At 
the same time, data collection by international organizations can be 
criticized, if insufficient attention is paid, in a balanced way, to the different 
social needs among the member countries. The globalization of the world 
economy, and the increasing global exchanges of educational services, 
knowledge, and technology transfers among nations increase the need and 
raise demands for Japanese education policies to fit into the international 
context. 

The science and technology of Japan will become more visible through 
efforts to provide information to international information data sets. This 
effort in itself will attract potential students and stimulate the undertaking 
of projects in international co-operation to which Japanese higher/ tertiary 
education can contribute. 

International efforts to develop strategic indicators are of great value. 
However, these activities have to be assessed from various social 
perspectives. What would be the impact of the introduction of standardized 
indicators among different social contexts? Would they genuinely support 
higher/tertiary education reform in the respective countries? Would they 
lead to fair competition among different social systems? These issues need 
to be cautiously examined. 

In this article, we have briefly discussed the development of qualitative 
indicators for the policies for higher/tertiary reform in Japan. Clearly, the 
establishment of such indicators provides strong incentives for achievement 
within the system. At the same time, mechanical or bureaucratic 
approaches to the quantitative surveying of qualitative matters may not lead 
to substantial reforms. Rather, they may fall into the trap .of mere 
improvement of the paper work. 

Indicators linked with specific objectives may certainly detect short-teiTn 
changes in higher/tertiary education. However, these indicators are not 
always suitable for long-term longitudinal study, since they are often 
derived from ad hoc surweys implemented to meet the shorl-temr needs of 
the government. For this reason, it may be difficult to persuade 
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governments to establish globally common and stable indicator sets that 

may not be directly related to specific domestic reforms. 

REFERENCES 

AraMOTO, A. “Academic Profession in Japan”, in, P. G. Altbach, ed. The 
International Academic Profession: Portraits of Fourteen Countries. 
Princeton: Carnegie Foundation for the Advancement of Teaching, 1996, 
pp. 149-190. 

Arimoto, a. “Recent Development of Higher Education Research and Higher 
Education Policy in Japan”, in, U. Teichler and J. Sadlak, eds. Higher 
Education Research: Its Relationship to Policy and Practice. Oxford: 
Pergamon, 2000, pp. 93-106. 

Asonuma, a. “Kokuritsu daigaku ni okeru kenkyu-hi hojo no pattern henka: 
‘tokutei mokuteki-ka’ to ‘kyoso-ka’” [The Changing Patterns of the 
Research Funding System of National Universities], Japanese Journal of 
Higher Education Research 2 (1999): 135-156. 

Boyer, E. L., Altbach, P. G., and Whitelaw, M. J. The Academic Profession: 
An International Perspective, Princeton: Carnegie Foundation for the 
Advancement of Teaching, 1994. 

Chuo Kyoiku Singikai [Central Education Council]. Kongo ni okeru gcUcko 
kyoiku no sogoteki na kaJciiju seibi no tame no kihonteki sisaku ni tsuite 
[The Basic Policies for the Comprehensive Expansion of School 
Education in the Future]. Tokyo: Central Educational Council, 1971. 

Kuroha, R. Shtnban sengo daigakuseisaJcu no tenkai [Development of 
Postwar Higher Education Policy]. 2’^'^ ed. Tamagawa: Tamagawa 
University Press, 200 1 . . 

Mext. Gakko kihon chosa houkokusho: koto kyouiku kOcan hen [Report of 
School Basic Survey: Higher Education Version]. Tokyo: Ministry of 
Education, 1948-2000. 

Mext. Golc/co kihon chosa houkokusho: shoto chuto kyouUcu kikan, senshu 
gakko, kakushu gakko hen [Report of School Basic Survey: Elementary 
and Secondary Education Version]. Tokyo: Ministry of Education, 1948- 
2000. 

Me>h'. Kyoiku shih.yo no kokusai hUcaku [International Comparison of 
Educational Indicators]. Tokyo: Ministry of Education, 1969-2000. 

Mext. DaigaJcu ni okeni curriculum to no kaUcaku Jokyo ni tsuite [The Current 
Status of the Curricula Reform in Universities]. Tokyo: Ministry of 
Education, 1999. 

National Institute of Science and Technology Policy. Indicators for Science 
and Technology-2000. Tokyo: MEXT, 2000. 

Research Institute for Higher Education, Hiroshima University. Koto 
kyoUcu tokei data shu: shiscdcu ban [Statistical Data for Higher 
Education. Preliminary Edition]. Hiroshima: Hiroshima University Press, 
1989. 



BEST COPY AVAILABU; 



172 



A.AR1M0T0, A. YONEZAWA, H. FUKUDOME, ondM. TAKEUQHI 



Research Institute for Higher Education, Hiroshima UNivERsny. Koto 
kyoiku tokei data shu: dai ni ban [Statistical Data for Higher Education]. 
2"*^ ed. Hiroshima: Hiroshima University Press, 1995. 

UNESCO. Higher Education in the Century: Vision and Action. World 
Conference on Higher Education. Paris: UNESCO, 1998a. 

UNESCO. Framemor/c for FYiority Action for Change and Development of Higher 
Education. World Conference on Higher Education. UNESCO: Paris, 
1998b. 



XIII. The Perspective of the United States 

Regarding System-Level Indicators for Higher 
Education 

JAMIE P. MERISOTIS 

This article briefly covers two areas. The first has to do with how the United 
States of America collects and disseminates information and data about 
higher education and how it develops its indicator systems. Secondly, it 
embodies a proposal relative to the organization of indicators for purposes of 
public dissemination. 

1. HIGHER EDUCATION DATA COLLECTION IN THE UNITED STATES 

The American system of indicators is very large, the reason being that the 
American higher education system is itself very large. There are over 3,300 
degree-granting colleges and universities and approximately 4,000 non- 
degree-granting colleges and universities in the American post-secondary 
system. More than 15 million students are enrolled per year in American 
higher education, about 78 percent of those in public colleges and 
universities, and about 22 percent in private colleges and universities. The 
two-year sector of higher education is very large. Almost 50 percent of all 
the students enrolled in what is called “post secondary” or “higher 
education” institutions in the United States are in these two-year 
institutions. It is therefore not surprising that the United States has an 
array of indicator and data collection systems in play at a variety of levels. 

The United States system is not - in any way, shape, or form - a national 
system of higher education. It is decentralized in many dimensions. There 
are a variety of data collection and dissemination mechanisms in play at the 
Federal, state, and local levels. 

The Federal government plays a very large role in terms of data collection 
and dissemination in the American education system. The largest system 
that is in place in the United States is called the Integrated Post-Secondary 
Education Data System (IPEDS). IPEDS is a census survey of institutions, 
being a multi-part survey of all post-secondary institutions. The data 
available in its data bank is institutional level data. Institutions that fail to 
report data to this system are denied participation in Federal student aid 
programmes in the United States. This measure is taken to enforce 
compliance in terms of data reporting and, as a result, virtually all 
institutions do report to this system. IPEDS provides data in a variety of 
areas, including enrollment, completions, faculty salaries and rank, finance 
data, and extensive data on expenditures and revenues in higher education, 
libraries, and staff. The IPEDS data system is a very large data system that 
is easily found on the Internet. One.nrriay download unrestricted and 
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restricted data that are available for purposes of private analysis and that 
go back to the mid-1980s. 

Another system in place is smaller, yet receives a great deal of attention. ' 
It is called the Student Right to Know Act, put in place in part because of a 
lack of accurate information about graduation and completion rates in 
American higher education. The only institution that published this kind of 
data before this law was passed was the National Collegiate Athletic 
Association (NCAA), which provided data on student athletes because of 
their large presence in American education. The collection of data on 
student athletes in turn caused the federal government to decide that it 
should be collecting the same data on all students in higher education. As a 
result, institutions are required to annually submit and disclose 
undergraduate completion and transfer rates. Again, these data are census- 
level data available from the institutions. The same penalties apply for lack 
of compliance. 

There is also an enormous array of federal sample surveys that deal 
mostly with student-level data. Faculty-level and household-level 
information are available in these various surveys and deal with issues 
ranging from student financial aid and student finance to completion, 
beginning enrollment, and a variety of other matters. In these data sets, the 
data record is the individual student, so data analysis is possible for 
tracking or tracer survey purposes, based on student records. 

The states also have a variety of data collection and dissemination 
systems. Performance reporting is proliferating in general. More than half of 
the states have some type of performance reporting system in place for the 
public colleges and universities and, in some cases, for the private colleges 
and universities located in them. The performance reporting systems report 
combinations of institutional and system-level data related primarily to the 
effectiveness and the efficiency of the operation of higher education in the 
given state. These systems vary significantly across states. Some have very 
large and complex data systems, with fifty, sixty, even eighty indicators that 
need to be reported by institutions. Other states, however, have only limited 
systems and only use five or six indicators. 

In recent years, the publication of a report titled Measunng Up 2000 by 
the National Center for Public Policy and Higher Education (NCPPHE, 
2001), received enormous attention in the United States. It aggregated 
existing data and developed its own indicators on an individual-state-level 
that examined five areas of performance of higher education in each of the 
fifty states. These areas include preparation, participation, affordability, 
completion, and benefits. This report captured the attention of American 
policy-makers and legislators. The unit of analysis in this system is not the 
individual institution, but only aggregated, state-level data. In other words, 
the NCPPHE tries to measure the performance of the given state itself 

A variety of other systems also exists. The systems are sponsored by 
private entities, including NGOs, polling organizations, and research 
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centers. The ranking systems that have proliferated in the United States 
could be included in this category. 

The American system poses several problems in terms of developing 
adequate indicator systems. One is that there is a cacophony of information 
uses and purposes for these data. We have so much data from so many 
different sources that it is quite confusing and in some ways can be 
deleterious to the policy development process. In addition, it takes a long 
time to develop and publish indicators, particularly at national level, as a 
result of this cacophony of information. A good example of this phenomenon 
can be seen in the case of The Condition of Education, published by the 
United States Department of Education. The most recent issue (2001) 
reports data from 1998 or earlier. Such data is very difficult to use for 
policy-making purposes because of the rapidly changing nature of the 
higher education enterprise. The market orientation of higher education is 
as much a characteristic of the United States as it is of most other 
countries. The use of technology, multimedia tools, etc., is significantly 
changing how higher education is delivered. These data systems are not well 
suited for measuring these kinds of changes. 

A second problem is that the American system lacks integration and 
coordination. Data collectors in the United States fail to use the 
standardized OECD definitions for most of indicators. There is poor 
integration across the federal, state, and other systems. To same extent, 
this situation is understandable because of the strong bias against the 
development of national goals in American higher education. The single 
national goal that is universally recognized in the United States is access to 
higher education, and the support of access primarily through very large 
investments in student financial aid. Students receive approximately 40 
billion dollars each year from the federal government alone in student 
financial aid, but the lack of national goals strongly discourages the effective 
coordination and integration of data systems. 

There are also large and significant gaps in information that exist despite 
the overwhelming nature of data systems and innovative approaches. For 
example, it has been possible to obtain national-level data that look at 
differences between undergraduate students and graduate students in 
terms of the funding of higher education. The IPEDS data system does not 
allow the user to desegregate undergraduate and graduate expenditures 
and revenues, a situation that has veiy important implications for analysis 
and other issues in higher education. 

The Institute for Higher Education Policy, an independent NGO in 
Washington D.C., is currently completing a Congressionally mandated 
study of higher education costs in the United States (available at 
<http://www.ihep.com/Org.php>). It is over 300 pages in length and uses a 
very complicated methodology in dealing with public and private 
institutions. The report says virtually nothing, however, because of a 
problem with the data system, therefore it is veiy difficult to arrive at true 
costs. 
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2. A PROPOSED METHOD FOR ORGANIZING INDICATORS 

There is limited focus in the United States on international comparisons. If 
one goes to the United States government to find international comparisons, 
one will not find anything beyond information about the elementary and 
secondary education levels. Comparative information about the higher 
education level does not appear to be of interest at Federal level. There are 
very few connections made in the United States among international 
organizations, the absence of the United States from UNESCO for two 
decades being the most obvious example of such a lack. 

The goal of the proposal is to articulate what higher education does for 
policy purposes. It should not, however, be construed as an argument for 
specific indicators, but rather as a way of organizing, at least some of these 
indicators, in ways that can be useful for policy purposes. The primaiy 
interest is in what the system is producing, based on the investment that 
has been made. When individual indicators are placed out in the public 
domain, they often lack context, a situation that can be dangerous in many 
cases. The proposal refers to the development of a conceptual framework for 
organizing the information. This conceptual framework, published by the 
Institute for Higher Education Policy a few years ago, reflects the benefits of 
higher education; 

Table 1. Four categories of benefits 

Private economic benefits 

Private social benefits 

Public economic benefits 

Public social benefits 

Source: Provided by the author 

A reasonable number of indicators on an international or system basis, 
or on a cross-national or system basis, can be placed in each of the grids so 
as to articulate what higher education is doing. This kind of approach would 
allow different systems to use different indicators and to emphasize different 
types of benefits depending on the focus of the given system, while allowing 
for commonalties for organizing the indicators reported. 

Table 2 is an example from the United States, based on the 1998 report 
by the Institute for Higher Education Policy. It is a framework using 
primarily output measures, but which would work for inputs as well. Based 
on data collected, the Institute was able to analyze, in an American context, 
the information in each of the four areas. The private economic part of the 
framework is what has received by far the largest public policy attention in 
the United States, because of the focus on market orientation. The Institute 
was able to use data developed by this framework in the policy debate to 
show that participation in higher education has other benefits. Obviously, 
there are challenges that can be made to each of these measures in terms of 
data quality, reliability, etc., but it was very useful in broadening the 1998 
Higher Education Act discussions. This framework, that was helpful in the 
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specific case of the United States, could also be adopted in some limited 
cases in a cross-national context. 



Table 2. A Framework for the different types of benefits provided by different systems 
of indicators 



Benefits 


Public 


Private 




Increased tax revenues 


Higher salaries and benefits 


Economic 


Greater productivity 
Increased consumption 
Increased workforce flexibility 
Decreased reliance on government financial 
support 


Employment 
Higher savings levels 
Improved working conditions 
Personal/professional mobility 




Reduced crime rates 
Increased charitable giving/ community 
service 


Improved health /life expectancy 
Improved quality of life for offspring 
Better consumer decision making 


Social 


Increased quality of civic life 
Social cohesion/ Appreciation of diversity 
Improved ability to adapt and use 
technology 


Increased personal status 
More hobbies and leisure activities 



Source: Provided by the author 



This framework of benefits could help drive the development of 
indicators. Over time, the point might be reached whereby cross-national 
policy development could be linked to this benefits analysis in a way that 
would benefit all the nations involved. 
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XIV. Using System Indicators to Stimulate Policy 
Development 

HERB O’HERON 

In Canada, each province and territory has jurisdiction over educational 
issues. The provincial educational systems have developed in varying 
contexts and are designed in varying ways. As a result, even data 
describing the different education systems do not always conform to the 
same definitions. In short, Canada faces the same kinds of challenges in 
designing and using system indicators as UNESCO does in making 
comparisons among nations. Despite these challenges, it is clear that 
provincial governments in all parts of the country are indeed making 
increasing use of indicators to provide some measure of accountability to 
the public as well as to steer activities within the universities. 

The Council of Ministers of Education, Canada, along with Statistics 
Canada, are increasingly relying on indicator reports to help compare and 
contrast the ten different post-secondary education systems that exist 
across Canada. In early 1990, the Ministers of Education in each province 
and territory agreed to create the Pan-Canadian Education Indicators 
Programme (PCEIP) in order to develop statistical information on education 
systems in Canada. The report from this project begins by outlining a 
series of indicators that help set the context in which provincial systems 
operate. The Pan-Canadian Indicators then provide key profiles of system 
characteristics, including students, human resources, and finance. Only 
then do they move into output and outcome measures such as labour 
market experiences for various groups of graduates. 

There are also some emerging uses of indicators to augment the more 
established reports on system indicators. There are numerous examples of 
performance-based funding and performance contracts, including those 
recently negotiated between the Government of Quebec and universities in 
that province, but each of these new uses focuses on institutional rather 
than on system indicators. 

1. THE USE OF INDICATORS BY THE ASSOCIATION OF UNP/ERSITIES 
AND COLLEGES OF CANADA 

The Association of Universities and Colleges of Canada (AUCC) has also 
been producing a series of indicators in a publication. Trends: The 
Canadian University in Profile. Published every three years, this publication 
identifies some of the important enrollment, faculty, finance, and research 
challenges faced by Canadian universities, along with some key 
international comparisons. 
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The use by AUCC of indicators goes well beyond the Trends publication, 
and increasingly, the indicators employed to highlight the challenges 
universities face are forward-looking and policy-oriented. They highlight 
factors that will undoubtedly affect the universities in the years ahead. 
They are the kinds of factors that cannot be ignored as policy is being 
developed and change is being considered. 

This approach is the kind that Frans Kaiser (2003, in this volume, pp, 
31-35) outlines in “System-Level Indicators for Tertiaiy/ Higher Education: 
Some Notes on Requirements and Use”. Here, Kaiser has focused attention 
on the design and use of indicators to help manage change, to be forward 
looking, and to be action-oriented. He notes that there is growing concern 
with content validity and a need for both ease of calculation and of 
interpretation. And he goes one step further in describing the importance of 
not using the indicators in isolation, by pointing out that indicators need to 
be combined into a map that emphasizes relationships and flows. 

It was further suggested that “systemic indicators” should possess an 
additional important feature - they should stimulate the imagination of 
policy makers. 

Indeed, stimulating policy dialogue is one of major roles of the AUCC. 
The Association lobbies the Federal government - especially with respect to 
university research and student assistance, in which the Federal 
government plays a direct funding role. The Federal government also plays 
an indirect role in the broader support of post-secondaiy education by 
providing funds transfers to provincial governments that have 
constitutional jurisdiction over education. 

The AUCC has used indicators of enrollment demand to stimulate 
discussion about the expected level of university participation in the 
decade ahead. In this case, the target was broader than the Federal 
government. The messages were targeted at the member institutions of the 
Association of Universities and Colleges of Canada and to provincial 
officials with the goal of raising awareness and stimulating debate at all 
levels. 

Figure 1 below summarizes some of the factors exarnined to demonstrate 
that the demand for university education should grow strongly in Canada 
in the coming decade. Indicators in each of these areas were used to focus 
the accessibility policy dialogue around the interrelated goals of capacity, 
quality, and equity. It has been argued that with little excess capacity in 
the system, access to universities could not be maintained - let alone 
expanded - without adding the requisite human and physical resources. 
Without those resources, quality would suffer and increasing student 
numbers, without maintaining quality, would not represent improved 
access at all. 

To build consensus, AUCC began by assembling a number of indicators 
of factors that would most likely affect student demand in the decade 
ahead. The indicators included demographic change - the growing youth 
population and declining mid-career populations resulting from the baby 
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boom echo and the baby bust respectively - to help identify where student 
demand pressures were likely to occur. 



Figure 1. Demographics: Growth in the youth cohort 
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Family and peer 
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Labour market signals: 
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Improving student services Student aid: Institutions and government 



Source: Based on population projections of Statistics Canada (June 2001). 



Figure 2. Canada: Youth population (aged 18 to 21) 
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The indicator portrayed above shows that while the demographic 
pressures for growth are extremely strong in some parts of the country, 
other regions will face declines in traditional student populations. AUCC 
also demonstrated that shifts in participation rates could play a far more 
important role in enrollment change than demography. For example, as 
Figure 3 shows, during the 1980s, population in the most important 
student cohort fell by more than 20 percent, whereas the number of 
students coming from that cohort increased by 25 percent. 

Figure 3. Shifts in the enrollment and population cohort aged 18 to 21 




Source: AUCC based on Statistics Canada data (June 2001) 

It was important to highlight these different demographic and 
participation pressures to demonstrate to Federal government officials that 
a “one size fits all” policy will not gamer widespread support across the 
country. In a federal system like that of Canada, one-dimensional policies 
are unlikely to be effective or to be equally well received in all parts of the 
country. 

AUCC then looked at various factors that would likely affect participation 
rates. Indicators of shifts in labour market demand were examined to 
identify the types of signals that potential students might be receiving from 
the labour market. Over the decade of the 1990s, full-time job growth has 
been particularly strong in health, social science, education, government 
occupations, and natural sciences occupations. 

Moreover, these are the very fields that require the highest levels of 
education; some 30 to 70 percent of the employees in these occupations 
hold university degrees. 
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It is clear that demand for highly qualified personnel will continue to 
expand and that students are indeed responsive to such signals. 

AUCC looked at indicators of economic returns to examine the potential 
influence of income on the demand for university education. While it is 
generally recognized that earnings increase with age and experience, 
education plays a very important and under-acknowledged role in this 
equation. 



Figure 4. The most rapidly growing occupations require the greatest amount of 
education - 1990 to 2000 




□ Full-time job growth □ Pi*ojx>rtion with a university degree 

Source: Statistics Canada, Labour Force Survey (2001). 

Income increases early in the careers of most employees, but flattens out 
quite soon for those with little education, while it grows continuously for 
university graduates. Incomes of university graduates start higher and 
increase rapidly early in the career. Income expansion continues 
throughout most of the career. 

It is clear that the completion of a university education continues to be 
profitable - sending out yet another strong signal to potential students. 

There is, nevertheless, concern that today’s graduates will not enjoy 
these kinds of income profiles or premiums. However, some recent work by 
Statistics Canada dispels some of these concerns and strongly suggests that 
today’s graduates can reasonably expect similar earning premiums in the 
future. 

The next indicator examined the rather well-known relationship between 
income and university participation - those in the upper income quartile are 
twice as likely to have attended a university than those from the lowest 
quartile. 
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Figure 5. Income premiums increase with education and age 
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Source: Statistics Canada, 1996 Census 



While the data are now somewhat dated, Figure 6 indicates that while 
participation grew for children from all income backgrounds, the gap 
between the poor and the middle income groups expanded between 1986 
and 1994. A new study just released by Statistics Canada suggests that the 
“access” gap between those in the highest and lowest socio-economic groups 
in 1998 remains quite similar to the 1994 levels. 



Figure 6. Family income plays an important role in university participation 
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□ 1986 □ 1994 



Source: Statistics Canada, General Social Suivey (1994). 
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More importantly, these kinds of access indicators show that there is 
significant room for improved equity in university participation. AUCC has 
been, and continues to be, a strong advocate of enhanced student 
assistance and improved debt reduction measures to help individuals meet 
the full range of costs associated with seeking higher education. Over the 
last decade, Canadian universities have used some of their additional 
tuition revenue to more than triple their own scholarship and student aid 
programmes. Various forms of student aid combined with an expansion of 
the capacity of universities to encourage and meet enrollment demands will, 
together, create the conditions for expanding equitable access. 

AUCC looked at indicators of the influence of parental education on 
today’s youth (Figure 7). This indicator shows that on average it is likely 
that children in families in which the parents have attended a university 
have an advantage that goes beyond the financial support that such parents 
may be in an enhanced position to provide. These parents may also create a 
home environment that, on average, not only stimulates an enhanced 
attitude towards learning, but also encourages an increased understanding 
of the importance of post-secondaiy education and of the need to earn the 
grades to gain access to it. 

Figure 7. Canada: Probability of completing tertiary education increases with the 
level of education of parents. 
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Source: AUCC using OECD data. 

This indicator shows that many more of the baby boomers of today than 
adults in earlier generations have completed university degrees. When 
combined with the previous indicator, it helps demonstrate that the family 
environment will likely lead to greater social pressure on young people today 
to attend a university. Once again, another indication is provided that the 
demand for university education should expand over the coming decade. 
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Figure 8. The growing educational attainment of parents should push up youth 
participation 




SoiU'ce: Statistics Canada. 

System differences continue to make it difficult to interpret international 
participation rates. However, it is obvious that Canada is far from being a 
world leader in terms of university participation. Many nations surpassed 
the achievement of Canada in this regard during the 1990s, and others are 
closing previous participation rate gaps (Figure 9). 



Figure 9. Nations having surpassed the Canadian level of university participation, 
1998 
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Source: OECD Database, 2001, 
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This indicator also demonstrates the degree to which developed nations 
recognize the importance of university systems in becoming more 
knowledge-based. It also clearly highlights the need in Canada to establish 
targets for participation across all levels of the post-secondary system. For 
Canada to be among world leaders, its participation rates will have to 
increase significantly. 

Figure 10. Projected full-time enrollment in Canadian higher education 
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Source: The author. 



Combining the demographic indicators with the participation indicators 
leads to the projection of a very strong increase in student demand in the 
coming decade. However, even at the highest projected level, less than one- 
quarter of young adults will be enrolled in Canadian universities in a decade. 
Such trends are consistent with past growth in participation and other 
projections of the demand for university graduates on the labour market. 

One of the most important implications of the expected enrollment 
growth is the impact it will have on faculty hiring requirements. If 
universities are to have the capacity to provide quality higher education to 
growing numbers of students, they will need to expand their human 
resources quite rapidly to match the expected enrollment expansion in the 
coming decade. 

Universities are clearly not very well placed to enroll many more students 
without hiring more faculty members to teach them. Over the last twenty 
years, enrollments have increased by 60 percent, while faculty numbers 
expanded by only 8 percent. In fact, faculty numbers are now 12 percent 
below what they were in the mid-1990s. Even if faculty growth keeps pace 
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with enrollment growth, over the coming decade, none of that gap will be 
closed, and the ability of faculty to enhance the quality of their programmes ‘ 
will be jeopardized. 

Figure 1 1 . Growing pressure for renewal: Fewer faculty members to support more 
students 
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Source. AUCC based on Statistics Canada data. 

However, growth is not the only detenninant of hiring needs. 

Owing to the age profile of faculty, replacing retiring faculty will also 
become a growing part of the need to hire over the coming decade (Figure 12). 

Figure 12. Movement of the demographic bulge through the system 




Source. AUCC based on Statistics Canada data. 
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Figure 13. Canada ranks fifteenth among OECD nations, in GERD to GDP ratios 
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Source: OECD, Main Science and Technology Indicators. 

Finally, the Federal and many provincial governments have targeted 
recent investments on research conducted in AUCC universities. In the last 
few months, the Federal government has articulated a goal to move the 
relative research performance of Canada from fifteenth to fifth place 
internationally. According to the projections of AUCC, the realization of this 
goal would likely require a tripling of the total investments in research and 
development (R&D) of the nation (see Figure 14). 

Figure 14. Need for 30,000 faculty members in Canadian Universities by 2010 




Source: Association of Universities and Colleges of Canada. 



This R&D goal therefore has major ramifications on the demand for 
highly qualified personnel, both within academe and in other sectors of the 
economy. It clearly means becoming more knowledge-based, which would 



190 



H. O’HERON 



drive up labour market demand for university graduates at an even more 
rapid pace than dictated by recent labour market trends. 

Taken together, these groups of indicators suggest that major changes 
are on the immediate horizon for Canadian universities. The policy options 
required to meet these emerging challenges will be many and varied (Figure 
15). The challenges and responses to them will likely vaiy from province to 
province and institution to institution. The focus in some provinces may be 
on improving quality, while the need to expand rapidly to meet student 
demand may cause other provinces to focus their efforts on capacity 
building measures - both human and physical capacity. 

In any event, the benefit of using the indicators in this way has helped 
focus the policy dialogue around a few key questions: 

- How many students and which students should have access to 
Canadian universities, colleges, trade, and other training institutions? 

- What human and physical resources are required to meet the rising 
demand? 

- How can quality be maintained or enhanced as capacity is expanded? 

- How can alternative delivery mechanisms be harnessed to achieve the 
twin goals of improving both access and quality? 

Figure 15. Key policy implications 
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2. FOCUS ON STRATEGIC CHOICES 

Thinking strategically involves an attempt to move an agenda forward, often 
by focusing efforts on a limited number of issues. However, as Guy Neave 



USING SYSTEM INDICATORS 



191 



(Neave, 2003, in this volume, pp. 63-74) points out, there are risks involved 
in this approach - not the least of which is short-term gain for long-term 
pain. For this reason, it is important to use system indicators that look 

further into the future than the end of the mandate of the current 

/ 

government. 

Using indicators in this way will help stimulate the policy discussion arid 
refocus stakeholder interactions on truly strategic goals and priorities. It is 
not really the indicators that are strategic; rather, it is the policy 
development and recommendations for change that ai'e strategic. Strategy is 
or should be both context and time sensitive. What is important today in 
one education system may not be nearly as important to address ten years 
from now, nor will all indicators be equally important in all countries. Too 
strong a focus on a common set of goals and indicators across all higher 
education systems is likely to lead to problems of conformity as outlined in 
other studies in this volume. 

Being strategic also implies the ability to move an agenda forward. It is 
difficult to imagine how one would be able to develop a successful strategy 
to simultaneously advance the complex and lengthy list of goals in the 
UNESCO recommendations for Higher Education in the Twenty-First 
Century: Vision and Action Plan (UNESCO, 1998). Rather, it is more likely 
that goals will need to be prioritized and adjusted over time. Quality in 
higher education is quite subjective, and improvement can always be made. 
But, improvement and change occur in a step-like fashion. Trade-offs 
abound. While we would hope to be making constant improvements, the 
opportunity to make changes will vary. 

Once partially addressed, relative priorities shift and new priorities are 
identified. This situation could well lead to the other major pitfall that Guy 
Neave highlighted at the outset of his article - an overemphasis on short- 
term objectives, driven by increasingly foreshortened planning horizons. 

Moreover, disciplined by the harsh lessons of prolonged recessions, 
governments today will only rarely commit themselves to long term 
spending requirements. While we may not like this fiscal context, it is 
imperative that we learn to work within it. 

3. CONCLUSION 

The indicators that have been used are meant to be forward looking so as to 
focus attention on future system needs. They were designed to stimulate 
policy development. Process indicators or indicators of the quality of 
provision have been avoided. Most of those indicators cross over into the 
realm of institutional rather than system-level indicators. 

Diverse institutional reactions to change should be expected and 
encouraged. The diversity represents a richness from which we can all draw 
when we are tiylng to assess what kinds of policies and processes work for 
our different student clienteles. These are the kinds of Indicators that 
institutions need to use to help identify areas within them that could be 
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improved or changed, but they are very difficult to generate and interpret at 
system level. 

Indicators, either at the system or institutional/ programme level, must 
be used with great caution for they seldom provide evidence of cause and 
effect relationships. They are no substitute for sound reasoning and good 
judgment. In the end, their primary purpose is to help identify questions 
rather than to provide aU the answers. 
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Education 
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1. INTRODUCTION 

This study addresses three matters in addition to providing a overview of 
the Australian Higher Education system: (i) It reports on the nature of 
system-level performance indicators in Australian higher education, (ii) It 
offers a view on their current status and reports on issues related to their 
design and use. (Hi) It offers a personal perspective on the UNESCO project 
and on the potential for an Australian contribution to its implementation. 

Being necessarily brief, this study cannot possibly do justice to the 
wealth of detailed information and commentary that exists about system- 
level performance indicators in Australian higher education. The 
publications and data sources cited in the paper represent the most 
significant contributions to date, but, no doubt, further data could be made 
available. 

2. AUSTRALIA AND THE AUSTRALIAN HIGHER EDUCATION SYSTEM: AN 
OVERVIEW 

Australia is an island continent stretching 3,180 kilometers from North to 
South and 4,000 kilometers from West to East. It is a relatively arid 
continent, with 80 percent of its landmass receiving a median rainfall of 
less than 600 mm per year, and it has a wide range of climatic zones: 
tropical regions to the North, arid expanses in the interior, and temperate 
regions to the South. Its population, which in June 1999 was just under 19 
million, is concentrated heavily in the major cities of Sydney, Melbourne, 
and Brisbane, and along the eastern coastal fringe of the continent. Its 
economy is robust and the standard of living enjoyed by most Australians 
compares favourably with that of other industrialized economies. 

It has a federal system of government within which there are three 
levels: Commonwealth, state, and local. At Federation, in 1901, the 
governments of the six Australian states assumed major responsibilities for 
education. While this situation remains, the Commonwealth government 
has become increasingly important as a funding source, especially for 
universities. 

There are thirty-seven publicly funded and two private universities in 
Australia. These institutions range in size from approximately 41,000 
students (Monash University, near Melbourne) to approximately 2,500 
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students (University of the Sunshine Coast, located just north of Brisbane). 
The total student population is approximately 680,000. 

In aggregate, Australian universities offer programmes of study across 
all fields found in universities in industrialized countries. Most students 
(78 percent) are enrolled in undergraduate programmes. These 
programmes, typically of three or four years in duration, include 
foundation programmes in the sciences, business, and the humanities, as 
well as professional programmes in fields as varied as medicine, 
engineering, psychology, law, nursing, and education. The proportion of 
students (22 percent) enrolled in postgraduate programmes is steadily 
increasing. These students are enrolled in postgraduate Diploma, Master’s 
Degree, and Doctoral programmes. 

Most students (59 percent) study full-time, especially at the 
undergraduate level. A substantial proportion (27 percent) of students 
study part-time, and a smaller but growing proportion (14 percent) of 
students are enrolled externally, that is, they do not participate in on- 
campus studies. 

State governments established almost all of the universities of Australia 
and gave each a mandate enacted by the relevant state parliament. The 
states have long ceased to be the source of funding for Australian 
universities, however, and the Commonwealth government is now the 
principal funding source, providing 68 percent of their income in 1999. 
Other sources include student fees and charges, research grants, income 
from investments, income from the state governments, and income from 
donations and bequests. As much as one-third of the Commonwealth 
government income is received through the Higher Education Contribution 
Scheme (HECS) of the Commonwealth government. Under this Scheme, 
students who have a Commonwealth-funded place must make a 
contribution (estimated at one-third) to the cost of their studies. They do 
this either by making an up-front payment or by defeiTing payment (plus 
accumulated interest) until their income reaches a certain level, at which 
point they repay the Commonwealth through the income tax system. 
Because overseas students do not have access to Commonwealth-funded 
places, they do not pay into HECS and instead must pay full fees. 

Successive Commonwealth governments have sought to contain the 
public cost of universities, either by charging more to students through 
HECS or by encouraging universities themselves to obtain, more of their 
funding from non -governmental sources. Not surprisingly, funding is one of 
the most important current concerns of Australian universities. Through 
the Australian Vice-Chancellors’ Committee, universities are seeking to 
persuade the Australian public that increased public funding for higher 
education is necessary for them to maintain international standards and to 
retain their high reputation for teaching and research. 

Australians have high levels of participation in post-school education 
and training. About one-quarter of all Australians hold a post-school 
qualification. Recent OECD (1998) figures indicate that Australia ranked 
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third, after Canada and the United States for tertiaiy level participation 
among those aged 17 to 34, and seventh, after Spain, Finland, the 
Netherlands, the United States, Canada, and Denmark, for university-level 
participation. 

3. STATEMENT OF PURPOSE OF THE AUSTRALIAN HIGHER EDUCATION 
SYSTEM 

The principal purposes of Australian universities (Higher Education 
Council, 1992, pp. 12-13) are: 

- the education of appropriately qualified Australians to enable them to 
take ...leadership role[s] in the intellectual, cultural, economic, and 
social development of the nation and all its regions; 

- the creation and advancement of knowledge; 

- the application of knowledge and discoveries to the betterment of 
communities in Australia and overseas. 

Australian universities, wherever their location and whatever their 
selected profile, must enable their graduates to operate anywhere and in 
any sphere at a level of “professionalism” consistent with best international 
practice, and in ways that embody the highest ethical standards. 

4. THE GOALS OF THE AUSTRALIAN HIGHER EDUCATION SYSTEM 

The aggregated goals of the system are, accordingly, to serve the 
community by; 

- retaining and nourishing its own diversity to meet the needs of a 
nation that is characterized by its geographical, social, and cultural 
variety; 

- encouraging further diversity so that all courses reflect the regional, 
social, and cultural differences that impinge on the individual 
universities; 

- providing a range of opportunities for access to higher education by 
members of disadvantaged groups; 

- ensuring high quality programmes of teaching and supervision; 

- ensuring that graduates of Australian universities are enabled to 
operate anywhere in Australia or overseas at standards consistent 
with best practice; 

- ensuring that their students are encouraged to achieve beyond their 
own expectations; 

- providing an intellectual climate within the institutions that 
encourages the questioning of currently accepted knowledge and 
modes of inquiry, its foundations, and its presuppositions; 

- achieving scholarly depth and perspective in matters relating to 
society, technology, and culture; 

- advancing knowledge through research that is, at its best, the equal 
of any that is conducted in higher education systems elsewhere; 
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- collaborating with other teaching and research sectors, such as 
schools, TAPE,* and government research agencies, and with 
industry, to ensure the most effective use of resources; 

- applying the outcomes of research and scholarship in ways that 
contribute to the economic, social, and cultural development of 
Australia, its states and regions, and its place in the world; 

- engaging actively with the professions and in the continuing 
education of practitioners; 

- keeping the wider community abreast of developments in their 
selected areas through participation in community debates; 

- engaging with the community generally, as educator and as social 
and cultural critic at local and national levels, according to the 
capacities of the individual institutions; 

- providing the Australian community generally, but students and staff 
in particular, with access to the most recent advances in knowledge, 
and its application, through relevant international networks 
complementing those of the individual institutions. 

5. STRATEGIC SYSTEM-LEVEL PERFORMANCE INDICATORS IN 
AUSTRALIA 

There has been sustained interest in performance indicators for higher 
education in Australia since about the mid-1980s. The development of 
indicators since then has been striking. They now form an important basis 
for monitoring both the performance of the Australian higher education 
system and the performance of its constituent institutions. Their 
importance to date as a basis for Commonwealth government funding 
allocations has, however, been restricted. 

While there are numerous references to performance indicators in 
documents on Australian higher education written prior to the mid-1980s, 
the first significant step towards their coordinated identification and 
development was taken in 1984, when the Commonwealth Tertiaiy 
Education Commission funded a study group to investigate ways of 
measuring quality and efficiency in Australian higher education (Linke et 
al, 1984). The group pointed to a range of possible performance indicators 
for the higher education system, but its conclusions were cautious. In 
particular, it expressed concern that system-level indicators might be 
“insensitive to local or circumstantial conditions” when applied to 
individual institutions (Linke et al, 1984, p. iv). The implied reason for 
caution was, at least in part, the distinction between quantitative and 
qualitative performance indicators. 

The Commission pursued the matter. In a report to the Commonwealth 
government in 1986, it strongly supported moves to improve the range and 
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availability of statistical information on higher education in Australia, and 
it encouraged the development of performance indicators. It again 
expressed caution. In its view, the overarching goals for the system had not 
been agreed upon, and the management information system available at 
the time was considered to be inadequate to the task of yielding an 
assessment of the achievements of the system (Hudson, 1986, pp. 268- 
270). 

In 1988, the Commonwealth government announced its intention to 
develop performance indicators that would permit measurement of the 
achievement of explicit strategic goals for higher education. The Minister 
stated that 

The range of indicators to be developed should cover such issues as 
student demand and course completion rates, quality of teaching 
and curriculum design, relative staffing provision and measures of 
academic staff performance in various aspects of research, 
publication, consulting, and other professional services. Indicators 
of performance against equity goals and measures of organizational 
efficiency should also be included in this process (Dawkins, 1988, 

p. 86). 

These specifications laid the groundwork for subsequent developments. 

In 1989, the Commonwealth government commissioned a research 
group to develop performance indicators for the system. Twenty-seven 
generic indicators were developed, including indicators of institutional 
context, teaching and learning, research and professional services, and 
participation and social equity (Linke, 1991). The work of this group was 
invaluable for' the integrated nature of the set of indicators produced and 
for the originality of some individual indicators. The focus of the work 
undertaken by the group, however, was upon developing indicators that 
would be of primary value to individual institutions. Indeed, the group 
recommended that the indicators 

be used by institutions as a basis for setting appropriate achievement 
targets - taking account of the range in performance of comparable 
institutions across the system as a whole - in accordance with their 
particular goals, resources, and other local circumstances (Linke, 
1991, p. 134). 

Little emphasis was placed on the potential usefulness of the indicators 
as a basis for monitoring system-level performance. 

In 1990, the Commonwealth government introduced for the first time a 
performance-based scheme for allocating funds to higher education, having 
foreshadowed this possibility in 1988 (Dawkins, 1988, p. 86). The scheme 
that was introduced confined itself to the funding of research 
infrastructure, however, and it did not include the range of indicators 
articulated earlier by the government. It required universities to compete 
for a share of funds (approximately 5 percent) that were held back from the 
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total operating grant to universities. The basis of this competition was 
institutional performance on an index of income obtained from national 
competitive research grants. 

The scheme remained in place throughout the 1990s, with modifications 
made in 1995 to broaden the index to include indicators of research 
outputs. In 2002, it will be replaced by a new performance-based scheme 
for funding both research infrastructure and the training of higher degree 
research students. As a consequence of these changes, up to 15 percent of 
public funding for Australian universities will eventually be performance- 
based. Of particular note is the focus upon performance indicators. 

Over the period from 1993 to 1995, an extensive quality review process 
was introduced into Australian higher education. Curiously, the audit 
process initiated and funded by the Commonwealth government did not 
require systematic use of quantitative performance-related indicators. As a 
result, universities produced individualistic quality portfolios based upon 
their particular interpretations of quality as reflected within their 
institutions. Portfolios varied considerably in their assignment of values to 
performance indicators. The modest financial allocations made by the 
government in recognition of quality assurance measures encouraged most 
universities to develop their management information systems and their 
skill in using performance indicators. 

In 1996, the Commonwealth government signaled that quantitative 
perfonnance indicators would, when possible, provide the basis for future 
quality appraisals of higher education. Both the Department of Education, 
Training, and Youth Affairs (DETYA) of the Commonwealth government and 
principal advisory body of the government on higher education, the Higher 
Education Council, were directed to develop both institution-specific and 
system-level performance indicators for Australian higher education. 
Publications that reflect the renewal of government commitment to the 
importance of quantitative indicators include Equality, Diversity, and 
Excellence: Advancing the National Higher Education Equity Framework 
(Higher Education Council, 1996), The Characteristics and Performance of 
Higher Education Institutions (DEHYA, 1998), Equity in Higher Education 
(DEHYA, 1999a), and The Quality of Australian Higher Education (DEHYA, 
1999b: 1-22). 

To summarize, the Commonwealth government has succeeded in 
generating a large and diverse range of quantitative performance indicators 
for higher education in Australia. However, system-level indicators have 
not been developed to the same extent as institution-specific indicators. 
The reasons for this feature of the Australian experience are now 
addressed. 
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6. THE CURRENT STATUS OF STRATEGIC SYSTEM-LEVEL PERFORMANCE 
INDICATORS IN AUSTRALIAN HIGHER EDUCATION AND THE PROBLEMS 
OCCURRING IN THEIR DESIGN AND USE 

The current strategic priorities of the Commonwealth government for post- 
school education and training (including higher education) are to expand 
opportunities, assure quality, improve the responsiveness of institutions to 
varying student needs and industry requirements, advance the knowledge 
base and university contributions to national innovation, and ensure 
public accountability for the cost-effective use of public resources (DETYA, 
2000, p. 35). Applied to higher education, achievements as per these 
priorities are measured by means of indicators that include “access to 
higher education”, “participation of the population in higher education”, 
“higher education completion, retention and progress”, “destinations of 
higher education graduates” and “unemployment experience of higher 
education graduates”. In addition, the government monitors the financial 
health of the higher education system by means of indicators of income 
from all sources, expenditure for all purposes, and institutional liquidity 
positions. 

The list of system-level indicators is, however, limited. In particular, the 
quality of the performance of the system in discharging its core 
responsibilities of teaching and research is not comprehensively monitored. 
The issues surrounding appraisal of the quality of a higher education 
system and of its constituent elements were comprehensively documented 
in a report entitled Achieving Quality, which was published by the Higher 
Education Council in 1992. In short, the problem is one of measurement, 
referred to as 

the measurability of the attributes of quality, the problem of 
discriminating between change and improvement, and the difficulty 
of establishing which factors improve or impair the quality of 
outcomes (Higher Education Council, 1992, p. 72). 

In 1992, the Council documented a comprehensive statement of the 
purposes and goals of the Australian higher education system (Higher 
Education Council, 1992, pp. 12-13; v. Appendix B). In 1998, it embarked 
upon the establishment of relevant indicators (Higher Education Council, 
1998, p. 315). This project did not reach fruition, however, largely because 
of difficulties of measurement. The goal of “ensuring high quality 
programmes of teaching and supervision”, for example, requires not only 
input from recent graduates, as is currently obtained in Australia by 
means of the Course Experience Questionnaire instrument (Ainley and 
Johnson, 2000), but also input in the form of judgments by discipline- 
based peers. These different inputs are extremely difficult to integrate and 
to report in the form of a strategic indicator, and the process is very costly . 

There is, in addition, a view in Australia that “quality” is best measured 
in the context of the mission and circumstances of individual institutions. 
Linke, for example, referred to the fact that system-level indicators can be 
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“insensitive to local or circumstantial conditions” (Linke etal., 1984, p. iv). 
The Australian Committee of Vice-Chancellors stated that: 

the validity of the use of performance indicators depends on their being 
judged in the context of the special mission and role of the particular 
university concerned... they provide no basis for comparative analysis 
(quoted, in. Higher Education Council, 1992, p. 72). 

Whether as a consequence of this view or not, the thrust of much of the 
collection and analysis of performance-related data in Australia is 
undertaken for the purposes of producing indicators for the level of the 
given institution, rather that at the level of the system [e.g., DETYA, 1998). 

One factor that offsets to some degree the lower level of emphasis placed 
upon the development of system-level indicators in Australia is the relative 
strength of other forms of monitoring of the performance of the system. The 
Commonwealth government has the capacity to draw upon technical 
expertise and submissions from a range of community interests, either to 
review the system as a whole [e.g.. West, 1998) or to review particular 
aspects of the performance of the system. The government does not, 
therefore, wholly rely on quantitative system-level indicators of 
performance when seeking to monitor the performance of the system. 
Through its Evaluations and Investigations Programme, for example, it 
routinely commissions investigations of performance in areas of higher 
education that are not amenable to exploration by means of standard 
quantitative indicators. Examples include investigations of undergraduate 
and postgraduate teaching practices, flexible course delivery practices, 
quality management practices, the costs of different learning technologies, 
the commercialization of new knowledge, and ways of consolidating 
productive partnerships between higher education and industry. 

As reported earlier, performance-based funding is employed in a limited 
way by the Commonwealth government to allocate funds to individual 
universities for research infrastructure costs. In 2002, the scale of 
performance-based funding will increase substantially as the availability of 
funds for individual universities for the training of research higher degree 
students also becomes subject to performance in regard to particular 
indicators (research degree completions, research income, and research 
publications). The extent of reliance upon performance indicators in the 
research area sits oddly with the more general approach of the government 
to the funding of higher education, which is characterized by marginal 
adjustments to institutional grants based upon annual negotiations about 
forward student load. A review, funded by the Higher Education Council, of 
performance-based funding in the research area was far from encouraging. 
It reported that the mechanism introduced in 1990 “is contributing to a 
frenetic chasing of grants and publications which, in addition to its 
unintended effects on the quality of teaching and learning, must also 
militate against more reflective scholarship and the desires of academics, to 
keep abreast of broad developments in their disciplines” (Andefson et cd.. 
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1996, pp. 63-64). Although this situation has been widely recognized, it 
has not been addressed by the government to date. 

7. UNESCO AND AUSTRALIAN COLLABORATION 

The UNESCO World Declaration on Higher Education expresses a global 
vision for higher education. In seeking to monitor implementation of this 
vision in Member States, UNESCO requires a “quantitative, solid system of 
fact reporting” (from the “Project Outline” on Strategic Indicators for the 
Twenty-First Century). In a study intended to develop the issues, 
Abercromby and Fielden (2001) recommend various self-challenge 
questions and quantitative indicators that might be adopted by country- 
based higher education systems to monitor and to evaluate performance 
against the UNESCO vision. Their recommendations raise a number of 
considerations that are now addressed from an Australian perspective. 

The first point is that the Australian database on higher education is 
sufficiently well developed to be able to provide inforaiation related to most 
of the quantitative system-level indicators identified in the study by 
Abercromby and Fielden. Details related to student participation, staffing, 
expenditure, research and development, foreign students, the lifetime 
expectations of students graduating from higher education, and so on, are, 
for example, readily available from either the Department of Education, 
Training, and Youth Affairs, or from the Australian Bureau of Statistics. 
Some of the other proposed indicators, such as “gender inequalities in 
curricula and research”, are not so easily identified within existing data, 
suggesting that their meaning is, at the very least, in need of clarification. 

The second point is that the interpretation of each of the four World 
Declaration objectives referred to by Abercromby and Fielden (p. 8) is open 
to considerable differentiation from country to country - owing to the 
glaring contrasts internationally in the settings for higher education. This 
situation makes the task of articulating standard strategic indicators 
exceptionally challenging. In the case of Australia, for example, concern 
with the World Declaration objective of “promotion of access on merit” will 
require that attention be given to improving the success and retention rates 
of indigenous students in higher education. Yet a strategic indicator that is 
relevant to this imperative may be far too specific for application in other 
countries. It follows that the global vision for higher education of UNESCO 
may need to be contextualized within Member States as part of the process 
of its implementation. UNESCO might permit Member States to develop 
relevant proposals for how they will meet, and then indicate overall success 
in meeting, the policy objectives of UNESCO. 

The third point is that several of the self-challenge questions identified 
by Abercromby and Fielden (2001; 14-16) cannot be appropriately 

addressed by quantitative system-level indicators. For these questions, 
qualitative data and understanding are necessary. In Australia, the 
Commonwealth government, through its Evaluations and Investigations 
Programme, referred to above, commission a considerable number of 
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qualitatively based evaluations and investigations of system-level 
performance each year. For the UNESCO vision to be realized, quantitative- 
measures and indicators, along with agreed upon qualitative investigations, 
need to be enunciated and harmonized. 

A final consideration is that any compilation of standard quantitative 
system-level indicators for application internationally will need at least 
initially to be restricted in number and scope. The difficulty in obtaining 
agreed upon operational definitions for strategic indicators across 
separately funded systems of higher education is potentially great. The best 
available model for the task is the OECD publication. Education at a Glance 
(2001), which provides information primarily on the demographic and 
financial aspects of education across a large number of countries - but this 
model tends to understate the importance of the core functions of teaching 
and research in higher education. 
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CONCLUSIONS 



XVI. The Past and Future of Quantitative 
Indicators for Higher Education 



KLAUS HUFNER 



1. INTRODUCTION 

Having looked at the results of the World Conference on Higher Education 
of 1998, thereby having focused on the Framework for Priority Action for 
Change and Development of Higher Education (the Priority Action Plan) 
(UNESCO, 1998b), the first issue to be dealt with will consist of three 
questions; 

i) How many levels must be taken into account? 

ii) What are the specific methodological demands for the construction of 
indicators? 

Hi) What level-specific indicators should be constructed for what 
purpose? 

These questions are closely interrelated. It would be difficult to deal with 
them step by step, because they would lead to the question of whether or 
not it would be possible to construct a meaningful set of indicators without 
an implicit theoretical framework (which, in turn, has some political 
implications). In this context, it might be useful to identify the 
approaches/models that have been used so far in order to classify higher 
education systems. The case of the German higher education system is 
briefly described so as to reveal some of the difficulties of constructing 
indicators for a higher education system “in transition”. 

2. BACKGROUND AND TASKS AHEAD 

When the UNESCO World Conference on Higher Education adopted its 
Framework for Piority Action for Change and Development of Higher 
Education in early October 1998 (1998b), the document included a huge list 
of demands related to the main directions of reform and their governing 
principles. This set of demands, in turn, implied a conceptual framework as 
expressed in the seventeen articles laid down in Higher Education in the 
Twenty-First Century: Vision and Action (UNESCO, 1998a). 

Having studied the two UNESCO documents, as well as the papers 
prepai'ed for the meeting, the following picture emerges. 

The Framework for PHority Action is directed at three levels of priority 
action, at (i) the national level, (ii) the level of systems and institutions, and 
(Hi) the international (UNESCO) level. If one orders these levels vertically, 
one starts with the higher education institutions. Then one moves up to the 
(national) system level, perhaps introducing a regional level, as will become 
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more and more necessary in Europe (either the European Union and/or the 
European Higher Education Area). Finally, one reaches the global- 
(UNESCO) level. 

The Project outline stresses that “while the main responsibility for the 
process of implementation remains with respective Member States and their 
higher education institutions, UNESCO, in co-operation with other 
organizations and partners, is expected to follow up the developments and 
monitor the implementation” of the vision of higher education as presented 
in the two UNESCO documents cited above. 

Since UNESCO is a world organization with 188 Member States and 4 
associated members, but has at the same time regionally decentralized 
offices and institutes /centers, such as, for example, UNESCO-CEPES, it 
can be assumed that UNESCO is in charge of both, the intra- as well as the 
inter-regional task of monitoring. 

The usefulness of UNESCO monitoring might be questioned, because the 
two above-cited documents are neither obligatory, as in the case of 
conventions ratified by Member States, which also include optional 
protocols related to individual complaint procedures, nor do they include 
any timetables which indicate which goals should be reached in which year. 
Nevertheless, monitoring can become an effective tool when used in a 
comparative way, thus indicating which country has reached what level of 
implementation concerning the fulfillment of a specific goal. Performance 
ranking tables, for example, can be easily read. They do not need further 
detailed explanation - in shord, they represent a reduction of systemic 
complexity leading to vivid debates on higher education policy, especially in 
those countries ranking at the bottom. If the tables that are constructed are 
based upon the huge store of data available and published by UNESCO 
without larger time lags, UNESCO can play a major role in higher education 
policy-making. 

- As the participants in the Hiroshima Roundtable came frorn two 
different “sides”, from the institutional/national “side” and from the 
regional/intemational “side”, looking either “upward” or “downward”, 
it will be interesting to see how their different “views” can be 
reconciled or even merged, because the issue of the information 
aggregation level might cause many problems. 

Moving up the aggregation of the information level automatically implies 
that information will be destroyed. On the other hand, the more 
disaggregation occurs, the more information will be created. Indicators to be 
developed and statistically applied at the regional/intemational level will 
most probably be rather “cmde” measures, in fact, “proxies”' for the 
description of social realities. The selection of the number of indicators to be 
chosen therefore deserves special caution. Which are the best indicators for 
describing the performance of national higher education systems? 

In addition, another question must be raised, that of the need to 
constmct a new set of indicators which actually represent the 
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regional/global level - that, in other words, represent the stage/degree of 
globalization of higher education systems? One should think, for example, 
of input-output matrices of student and teacher mobility or even of “course” 
mobility, which will not only indicate which countries are “net” importers or 
exporters of higher education, but will also offer information about the 
“density” of the globalization reached, as well as about its directions, 
thereby also stimulating the undertaking of corresponding studies at 
regional level. 

Looking now at the national level as well as at the level of systems and 
institutions, one can differentiate between “externally” - and “internally” - 
oriented sets of indicators. In the first case, information is collected for 
international comparison. The information demand side consists of 
international organizations, for instance UNESCO, asking for such data for 
purposes of international comparisons. Most probably, most of the data will 
come from “internal” national data sets, some of which will have to be “re- 
analyzed”, and some of which will have to be collected through new 
enquiries and/or samples. 

Looking into the UNESCO document, one can find data requirements 
that are most probably available in many countries, such as, for example, 
indicators for 

- women’s participation at all levels in all disciplines: 

- national and international mobility of teaching staff and students; 

- low enrollment in higher education as per internationally accepted 
comparative standards. 

Other demands require specific research and agreements on how to 
define “proxies” and to measure them, such as, for example: 

- the exercise of academic freedom and institutional autonomy: 

- the use of multi- and trans-disciplinary approaches in teaching and 
research: 

- the recommendations and the new goals for (higher) education as set 
out in the 1996 report to UNESCO of the Delors-Commission, related 
to its four stated goals: “to learn to learn”, “to learn to do”, “to learn to 
live together”, and “to learn for life”; 

- accountability and both internal and external evaluation; 

- employability of graduates, not only as job-seekers, but also as job 
creators: 

- the use of new technologies in higher education; 

- the openness of higher education to adult learners. 

Admitting that this list is not comprehensive, the necessity of 
undertaking a full in-depth analysis of the two UNESCO lists remains a 
priority. Moreover, and here we return to the second question of the three 
questions cited at the start: What are the specific methodological demands 
for the construction of indicators? In this connection, the technical 
questions of empirical-statistical evidence as well as of the use and 
applicability of statistical methods are of secondary order. Putting the 
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matter more bluntly, the question is whether there is not a need for a theory 
or at least a model or taxonomy in order to start the work. 

What follows will offer some insights into how the problem was 
approached in the past. A new theoretical breakthrough will still remain to 
be undertaken. 

3. STATE VERSUS MARKET SYSTEMS - THE DEAD HAND OF THE PAST? 

Models existed and were used in the past in order to classify and to locate 
national higher education systems. Some of the implications of these 
models for higher education policy are discussed below. 

3. 1 . Three Categories for Locating Higher Education Systems 

Looking upon the ideal-type models of a continuum which links the two 
extremes of a central planning system, on the one hand, and a perfect 
market system, on the other, higher education systems could be placed - at 
least until the end of the Cold War - into three broad categories: 

- at the one extreme, the relatively highly centralized higher education 
systems of the fonner socialist industrialized countries, which were 
supposed to function as integral parts of centrally planned national 
economies, thus satisfying the necessaiy requirement for skilled 
manpower for their economies: 

- at the other extreme, decentralized higher education systems, e.g., in 
Canada and the. United States, operating within the context and 
under the rules of “free market economies”, in which higher education 
institutions resembled, as Martin Trow once put it, “the birth and 
death of small businesses in modem economies, and the patterns of 
success and failure of small capitalist entrepreneurs” (Trow, 1996); 

- the countries of Western Europe to be placed somewhere in-between, 
depending upon the relationships between publicly financed higher 
education systems and vaiying degrees of centralization/ 
decentralization - which is much more difficult to locate exactly in the 
continuum mentioned above than the two other categories. 

This third category, which nowadays also includes the countries in 
transition and many countries of the South, is the most interesting one, 
because the countries concerned are confronted with the issue of 
“deregulating” higher education systems in the context of increasing 
globalization in economic, technological, and cultural terms. 

Now, what does “deregulation” actually imply? Is there an “optional mix” 
between bureaucratic and market co-ordination mechanisms possible in 
order to avoid “state failure” as well “market failure”? Since “the market and 
the bureaucracy are not gin and tonic that can be mixed in any proportion 
wanted”, as Janos Komai (1992) once put it, we have to identify desirable 
patterns in which higher education systems can operate better than in the 
past. 
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In economics, we leam that the model of perfect competition demands 
the fulfillment of a number of conditions to be met in order to guarantee an 
optimal allocation of resources. Among them, there should be many 
participants on both sides of the market, on the demand as well as the 
supply side. Furthermore, the axioms of full market transparency and of 
economic rationality must be guaranteed. 

Everybody knows that reality looks different even if a trend towards the 
better use of market forces in higher education can be observed in most 
countries around the globe. 

In the past, as best reflected in the OECD literature in the 1960s 
through the 1980s, three higher education approaches dominated the 
relationship between governments and national systems of higher 
education. 

First of all, the manpower (forecasting) approach, which has been used 
as an educational programming technique, linked all the sectors of 
economic, social, and educational activities. The education sector, including 
higher education, was treated as a sub-system of central state planning 
which produced the necessary qualified labour required for the economy. A 
rather rigid relationship between the outputs of higher education systems 
and the necessary inputs for the economy were assumed. Of course, prices 
were also used within this programming technique, but their value as 
signals (“economic levers”) was part of the overall planning context and not 
a substitute for it. 

The manpower approach has also been used in a number of OECD 
countries; however, the most important difference has been the fact that in 
those market economies its use has been primarily an “academic exercise”. 
The political decision-makers were free to use it as an information base for 
decisions concerning higher education systems, thus either supporting the 
expansion or the contraction of certain higher education sectors, faculties, 
and course programmes in the light of anticipated, possible future 
requirements of the labour market. But the relationships remained 
extremely loose, because the labour markets, being much more flexible, 
followed a different set of rules. 

At the other end of the continuum, the rate-of-retum approach has been 
used, thereby implicitly claiming that educational programming should 
proceed on the basis of - individual and social - cost-benefit analyses in 
which the direction of price signals are of importance for educational policy 
- decisions. This approach, that has also been used in order to identify any 
labour market imperfections, is based on the assumption of a free and 
highly flexible higher education market, which functions as an automatic 
resource allocation mechanism. Here, the role of buyers (enrolled students 
in higher education institutions) regarding the character and quality of the 
product is strengthened, as is, indeed, the very character of the producer, 
namely the individual higher education institution. 

In this model, the notion of “planning” is only related to the institutional 
level and primarily implies the management of institutional problems in a 
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competitive higher education market system. For this reason, the competing 
producers are extremely active in producing and publishing a wealth of 
information concerning the performance of their institutions, and research 
on higher education plays a much more important, strategic function so as 
to meet the challenges of the higher education market in order to “survive”. 

Needless to say, the information needs and applied strategies of these 
two extreme models vary tremendously — both at institutional and at system 
levels. 

Turning to the third group of countries, their higher education systems 
are somewhere in-between the two extremes. During the 1990s, most of 
them decided to move towards the ideal type of “neo-classical world”, but 
their situations are still rather confusing. Regardless of the angle from 
which one views these systems, both are, internally and externally, in 
situations of dramatic mutation to be characterized by conflicting forces of 
dynamic change versus conservative stability/ stagnation. Higher education 
system change might take ten years or even longer. No matter whether the 
systems are “Byzantine versions of the market” or “Byzantine versions of a 
state” (Meyer, 1983), the most difficult problem to be solved here is the 
issue of identifying indicators during this process of transition during which 
the roles of neither the state nor the market are clearly defined. 

In the past, during periods of expansion and no major restraints, the so- 
called social demand approach dominated in order to identify a possible 
over- or under-supply of student places by relating anticipated or politically 
desired (or even fixed) outputs of the higher education system (“social 
demand”) with the supply side of the higher education market. Depending 
upon the inflexibility of the adaptation processes of the state -controlled 
higher education systems, the individual as well as societal risks of 
educational mis-investments were rather high. 

3.2. Burton Clark’s Triangle 

Another analytical approach, which relates in interesting ways to those in 
charge of applying one of the three programming techniques, has been 
Burton Clark’s famous triangle (Clark, 1983). In his analysis, he identified 
three major forces that influence the operation of higher education systems, 
namely the state authority, the academic oligarchy, and the market. In his 
“magic triangle” - where these three forces are angles indicating ideal 
characteristics of higher education types — the connection lines offer three 
dimensions: 

- Dimension 1: the state authority model versus the academic oligarchv 
model: 

- Dimension 11: the state authority model versus the market model: 

- Dimension 111: the academic oligarchy model versus the market 
model. 
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Each of these three dimensions can be discussed as a continuum 
between two ideal-type systems in order to locate national higher education 
systems within this triangle. 

Dimension 11 relates to coordination problems between the state and the 
higher education system within alternative planning set-ups (which were 
dealt with above intensively). More “market” and less “state” as a demand of 
educational policy has been discussed in the economic literature by looking 
at the continuum between the “Leontief world” of central planning systems, 
to be found in the former socialist industrialized countries, and the 
“neoclassical world”, with its model of perfect competition, which served in 
part as an orientation function for higher education systems, for example, 
in the United States and Canada. 

3.3. The Higher Education System of Germany - “Neither - Nor” 

Without doubt, the higher education system of Germany can stiU be placed 
firmly between the state authority and the academic oligarchy (Dimension 
I). Legal authority and regulations are firmly in the hands of the Lander, 
whereas - in particular, with regard to research in which academic freedom 
is the rule - the German system stands very close to the academic vertex. 
There are surprisingly few or no market forces operating in the system. 

Moreover, the German higher education system is remarkably resistant 
to change, partly owing to the lack of institutional autonomy, with deans 
and presidents serving as primus inter pares and lacking the power of chief 
executives. Although the demands of increasing competitiveness, of 
internationalization, of the introduction of the BSc and MA degrees, thus 
replacing the Diplom Degree, and of credit-point systems are slowly being 
implemented, no one knows what the system will look like in the future. For 
the time being, the German higher education system looks more 
“Byzantine” than ever before, because it is a “system in transition”, which, 
for the time being, cannot be properly described by a consistent set of 
indicators. 

Almost all political parties in Germany clearly and often express goals for 
higher education reform, yet the goals are not put into effect in terms of a 
new policy which would embrace key elements within a consistent higher 
education market model. 

Those goals are; 

- competitiveness both among national higher education institutions as 
well as among international ones, whereby the territory of fifteen 
European Union member states plays an Important role; 

- accountability and transparency, thus impl 3 ring the use of a full set of 
pertormance indicators: 



Economist Wassily Leontief (1906-1999) won the Nobel Prize in 1973 for his production analyses 
showing how changes in one sector of the economy can eiffect others. 
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- internal efficiency as well as effectiveness with regard to the needs of 
the labour market; 

- the highest possible level of quality output, both in terais of graduates 
and of research results. 

However, problems arise as soon as these goals are to be translated into 
an applicable reform model of higher education, which then necessarily has 
to be based upon a market-oriented model. Resistance comes from all 
circles because the implementation of such a model necessarily demands 
radical changes for all parties/stakeholders involved and engaged in higher 
education. 

Such a market model would imply “consumer sovereignty” on the 
student demand side, but, at the same time, a price for the services offered 
in tenns of tuition fees. It would also imply producer sovereignty in terms of 
differentiation and specialization of services at different institutions. 
Performance would be evaluated according to market-oriented decision- 
rules, and better performance would lead to higher income. The role of the 
State administration would change dramatically: instead of regulating all 
kinds of details down to the faculty and chair level through highly time- 
consuming administrative procedures, it would be primarily in charge of (i) 
framework legislation, (ii) direct - or better - indirect quality control through 
accreditation mechanisms, and (Hi) arrangements for basic funding. 

Since there is no general agreement among the major political decision- 
makers in Germany, the “Byzantine” nature of German higher education 
will create new Byzantinisms during its transition period. 

On the one hand, a hot debate is taking place regarding the introduction 
of student fees. The main feature of these debates is a typical German one 
concentrating on and distinguishing between the two extremes of 100 
percent pro or contra, thereby ignoring the whole stratum of possible 
solutions between the two extreme positions. The main reason is the lack of 
a consistent model of higher education reform, which would take both sides 
into account, namely, demand for and supply of higher education. Typically 
is the present legal requirement found in the 1976 Higher Education 
Framework Act (Hochschulrahmengesetz) that teaching performance should 
be (i) evaluated by students and (ii) lead to a differentiated scheme of 
payments for professors. According to basic economics, this kind of 
performance differentiation on the supply side of the market can only 
function in an effective way if both sides, students and teachers, are 
financially involved in the demand for and supply of teaching services. It 
would be unrealistic to assume that students can and will offer objective 
evaluations of the teaching performances of professors as long as they do 
not have to pay for these services. 

As in all higher education systems in which schemes of tuition fees exist, 
they must be closely linked to a functioning system of stipends (loans 
and/or grants). Secondly, the repayment conditions must be clearly defined, 
for example, through an academic fund model along the lines of the 
Australian model introduced in 1989, which requires repayment based 
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upon the future income of graduates. Thirdly, the income from tuition fees 
must be retained by the individual higher education institutions and should 
not lead to a reduction of the present level of public financial support. 

Two examples may illustrate the present situation in German higher 
education. The first one is related to the decision-rules of a market system. 
When higher education institutions introduced limited incentive schemes in 
order to improve performance, some faculties received extra money because 
of higher numbers of graduates and higher numbers of dissertations per 
teacher, more publications, etc. The implied decision rule would have been 
that the more active teachers would receive the extra money, if not 
personally, at least for the improvement of the technical infrastructures of 
their chairs. However, this way of proceeding was not the rule. Some 
faculties decided to allocate the money for library acquisitions, others 
divided the extra money on an average per chair allocation, and others even 
turned the economic decision-rule up-side-down by allocating the money 
among the “bad performers” under the assumption that those professors 
would then perform better. 

In other words, if one introduces monetary incentive structures among 
faculties based on performance indicators without defining the decision- 
rules to be applied at the chair/institute level, the whole incentive scheme 
may lead to “Byzantine” situations that defy the market model. 

The second excimple is related to the definition of a “student in a system 
without tuition fees. The German higher education system is still 
characterized by extremely long study periods for the first degree (about 6.8 
years). Many “students” register because of the social benefits they receive 
without seriously taking courses (probably, around 20-25 percent). As a 
consequence, the drop-out rate is rather high (about 30 percent and more). 
Given these circumstances - and everybody agrees, of course, that the 
situation must be improved — one wonders whether ony per student 
indicator is a meaningful proxy reflecting the internal situation in a higher 
education institution or system. 

Given only two examples, this author fully shares the views expressed by 
Barbara Kehm, when she summarized the 1999 transition situation of 
“Higher Education in Germany” in the following way: 

‘The ambivalent attitude, that can currently be noted, of many actors 
in higher education and higher education policy uis-d-uis issues of 
deregulation and differentiation is possibly best explained by the 
following four factors influencing not only debates and decisions but 
also actual changes; 

i) Concepts of market and differentiation continue to be rather 
unfamiliar to the German higher education system, and most actors 
have hardly any experience in terms of the respective procedures and 
processes. 

ii) There is no consensus concerning the character of indicators on 
which differentiation could be based. 
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Hi) So far, political actors in the field of higher education have not clearly 
indicated that they are aiming at renouncing the principles of legal • 
homogeneity and basic equality of all institutions of one type. 
Furthermore, there are no suggestions in terms of what might happen 
to the potential losers in a system of competition and differentiation. 
iv) The first attempts at institutional profile building currently taking 
place are still rather timid because no experiences with strategies of 
self-marketing and of niche-marketing have been accumulated 
Possibly, the recent UNESCO World Conference on Higher Education 
will prompt some large-scale efforts in this respect” (Kehm, 1999). 

4. PRELIMINARY CONCLUSIONS 

The UNESCO vision of higher education for the Twenty-First Centuiy is a 
normative statement of a number of goals and priorities. Some of them can 
be quantified in terms of indicators, because routine statistics on them are 
already in existence. Others demand further reflection and research in order 
to identify the appropriate “proxies” for them. 

Most higher education systems are in a period of transition from 
centrally planned state systems to more “deregulated” systems. During this 
period, which can also be described as a phase full of experiments and 
contradictions, it is extremely difficult to describe those systems through an 
appropriate set of indicators. 

In order to construct system-level indicators for higher /tertiaiy 
education, an analytical framework will be necessary before entering the 
phase of statistical - technical measurement problems. 

In the light of the transformation processes taking place in many 
countries towards market economies in an increasingly globalizing world, it 
wll be a useful starting point to analyze higher education systems as 
market systems wrth different degrees of deregulation. 

In addition to the construction of institutional and system-level 
indicators for higher education, not only comparative but also specific 
regional- and world-level indicators should be developed, which would 
reflect the increasing degree of globalization in higher education. 
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XVII. Suggestions Relative to the Selection of 

Strategic System-Level Indicators to Review 
the Development of Higher Education 

RICHARD JAMES 



1. INTRODUCTION 

This study examines the possibilities for selecting a strategic set of 
indicators for monitoring the system-level development of higher education 
by UNESCO Member States and their higher education institutions. It 
draws upon the international and national experiences of quantitative 
reporting about higher education. It discusses possible criteria for the 
selection of indicators and presents a preltminary framework of indicators 
for consideration. 

2. PURPOSE OF THIS STUDY 

The UNESCO “Strategic Indicators for Higher Education in the Twenty-First 
Century” project is related to two documents that resulted from the 
UNESCO World Conference on Higher Education (Paris 5-9 October 1998): 
Higher Education in the Twenty-First Century: Vision and Action, and the 
Framework for Priority Action for Change and Development of Higher 
Education. (UNESCO, 1998a; UNESCO, 1998b). 

The “Strategic Indicators ...” project is being coordinated by UNESCO- 
CEPES, Bucharest, Romania. The project involves three Invitational 
Roundtable meetings in 2001-2003, focusing on; 

- system-level indicators; 

- statistical indicators for the quality assessment of institutions; 

- indicators for institutional and programme accreditation. 

This study was prepared for the first of these roundtables, titled “System- 
Level Indicators for Higher /Tertiary Education”. Among the expected 
outcomes of this meeting was the formulation of a short list of strategic 
indicators. At the request of the meeting organizers, this study attempts to 
answer two questions; 

1. What lessons can be drawn from international and national 
experiences in order to improve quantitative reporting about the 
development of higher/ tertiary education? 

2. What quantitative indicators should be chosen in order to review the 
system-level development of higher/tertiary education in the context 
of the “World Declaration” and the “Priority Action Plan”? 
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The proposed indicators should facilitate further reflection on the 
development of higher education in the context of the vision postulated in 
the World Declaration” and the “Prioriiy Action Plan”. The objective of 
recommending system-level indicators arises from the belief that only 
through monitoring systemic as well as institutional development can the 
implementation of the speciflc recommendations of the World Conference on 
Higher Education be determined. 

3. LESSONS FROM THE EXPERIENCE OF INDICATORS IN HIGHER 
EDUCATION 

There is a considerable theoretical and practical foundation upon which this 
project can draw. For weU over a decade, work has been undertaken to 
develop reliable and useful indicators for higher education. The 
Organization for Economic Co-operation and Development (OECD, 1998; 
Cuenin, 1988; Kells, 1993) has produced cross-national studies on the state 
of development of performance indicators, and detailed critical analyses 
have been conducted in the United Kingdom (Cave etal, 1997; Johnes and 
Taylor, 1991) and other developed nations (McDaniel, 1996). In Australia, 
there has been extensive quantitative data collection at institutional level 
based on the work of Linke (1991, 1995). ^ Much of the analytical work has 
focused on the development of valid and reliable indicators for the purposes 
of institutional comparison within national systems, even if some 
universities have engaged in cross-national benchmarking exercises (James, 
1999).2 

The international interest in indicators has arisen because of pressures 
for both summative and formative evaluation. On the one hand, as higher 
education systems have expanded towards mass participation, governments 
have become concerned about university accountability and public 
transparency and have sought objective measures of the performance of 
institutions and national systems overall [e.g., Henkel, 1991). On the other 
hand, the interest in indicators has derived from the quality movement and 
its discourse of continuous improvement. In the latter case, some of the 
demand for objective measurement has been driven by the universities 
themselves in their pursuit of external reference points for charting 



Australia has established a system-wide indicator framework, based on the requirement that 
institutions annually provide the government with statistical data. The 1998 report by the Department of 
Education, Training, and Youth Affairs, The Characteristics and Perfonnance of Higher Education Institutions 
(DETYA, 1998) lists a large set of indicators under the titles of broad context, staff, finances, and outcomes. 
The development of indicators in Australia has occurred over a lengthy period and includes work carried out 
in 1991 by the Performance Indicators Research Group (1991), to test a broad range of quantitative indicators 
suitable for evaluating performance in higher education. This effort was followed by the release of two reports 
(DEET, 1994; DEET, 1996) by the Department of Employment, Education, and Training presenting various 
indicators for higher education institutions. The Characteristics and Petformance of Higher Ekiucaiion Institutions is 

available on-line, at: <http://www.detya.gov.au/archive/highered/otherpub/characteristics.pdf>. 

^ Universitas 21, for example, is a company incoriDorated in the United Kingdom with a network of 
eighteen universities in ten countries <http://www.universitas.edu.au>. An initial objective in the formation of 
the Universitas 21 network was to provide member universities with kindred partners for nurooses of 
international benchmarking. ^ ^ ^ 
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corporate development within commercially competitive, market-oriented 
contexts. 

Regardless of whether or not the purposes are accountability or 
continuous improvement, reliable measurement seems necessary for 
shaping judgments, policies, and actions. There remain well-known 
difficulties. These are not, however, entirely insurmountable for the present 
project as long as the limitations and bluntness of indicators are recognized 
and the sights of the project are set accordingly. Not all that is valued in 
higher education is easily measured and reported, and qualitative indicators 
are necessary for the important aspects of higher education that defy 
quantification. Even the aspects that can be quantified pose measurement 
challenges. Owing to the extent of system and institutional differentiation, 
lengthy lists of qualifications and caveats often accompany the simplest of 
indicators. In addition, the possible quantitative indicators form a vast 
matrix; Davis (1996) cited more than 300 possible indicators, while well over 
200 indicators are reported annually in Australia (DEHYA, 1998). 

The Invitational Roundtable was confronted with the formidable task of 
recommending a small set of indicators which would strike a balance 
between what is technically robust, valid, and reliable, and what is 
politically and practically possible and sensible. The recent national and 
international experience of indicators may help. Although it is a challenge to 
summarize what has been learned over ten to fifteen years of quantitative 
reporting of higher education, some key conclusions - or principles, perhaps 
- do stand out. 

1. Indicators that are abstract or based on complex formulae are not 
easily interpreted - or are misinterpreted - by the various 
stakeholders with an interest in higher education. Simplicity is a 
virtue, and indicators should be transparent and have good face- 
validity. 

2. Notwithstanding the value of simplicity, indicators that are 
excessively crude, have tenuous links to goals, or fail to detect 
subtle differences and changes over time are of little value and are 
liable to be disregarded. 

3. The most strategically useful results are achieved when an agreed 
upon framework for data collection and reporting is established. 
Particularly if comparative analysis is to be undertaken, indicator 
definition must be precise, and quantitative variables must have 
adequate psychometric properties. 

4. For most policy- related purposes, the number of indicators 
probably should be kept to a minimum; otherwise, data collection 
tends to become an end in itself. The experience of benchmarking 
activities conducted in the business world is that large numbers of 
numeric indicators create data collection and management issues 
that distract from analysis and the utilization of findings (Karlof 
and Ostblum, 1993). 
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5. While there are some “simple” or “absolute” indicators (Cave et al, 
1997), rarely are indicators neutral or value-free. The creation of. 
indicators establishes a hierarchy of values, and the act of 
measurement and reporting affects the object of measurement. 
Selecting indicators attaches privileges to certain goals and 
functions and, over the long term, may redirect resources 
accordingly. Whoever chooses indicators in effect determines the 
“right” direction in which to steer activities. The normative 
tendency of indicators is valuable in many circumstances, but 
can act against institutional diversity within systems. 

6. Categorization of indicators is helpful in making projects 
manageable. Most indicators in higher education assume a 
process or production model representing the university/ higher 
education as a system of inputs, processes, and outputs (Cave et 
al, 1997: 25-37). The output indicators {e.g., completions and 
completion rates, levels of satisfaction with courses and teaching, 
graduate employment rates, and destinations) are generally the 
most difficult to measure and require considerable subjective 
interpretation. 

7. Raw quantitative data usually require modification before they 
can convey sufficient meaning to guide policy or actions. The 
needed modification may involve representation as a dividend 
[e.g., public expenditure on education as a percent of GDP) or the 
calculation of a trend over time [e.g., annual percent change in 
public expenditure as a percentage of GDP). 

8. A characteristic of higher education is the lengthy time lag 
between actions and outcomes in many important areas. This lag 
creates particular difficulties for quantitative measurement if the 
purposes are continuous improvement. Analysis of indicators and 
any action planned as a result should take account of the time 
horizon for outcomes. 

9. The direction of measurement of some indicators is questionable, 
depending on whether they are assumed to be measures of quality 
or efficiency. Moreover, the interpretation of certain outcome 
indicators may benefit from analysis of output quality. If 
expenditure per student is taken as an example, according to one 
interpretation, high unit cost may be taken as an indicator of a 
high quality educational process. Low unit cost, on the other 
hand, might be construed as a measure of efficiency. In either 
case, any conclusions drawn will be of dubious value, unless data 
on the quality of educational outcomes is available. 

10. Measures of the quality of teaching and research and the value- 
added effects of higher education are particularly elusive and 
subject to the effects of reputation. The present performance 
indicators for teaching quality have alternative conceptual bases, 
and most have a highly subjective element to them (Cave et al. 
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1997; 225)3. McDaniel (1996) learned - from a large European 
opinion survey of interest and expert groups - that, of seventeen 
commonly cited process indicators, only seven were deemed 
relevant indicators of quality in higher education: (i) evaluation by 
employers of graduates: (ii) academic reputation of staff; (Hi) peer 
review of curriculum content; (ivj employment rate of graduates: 
(v) completion rates; (vi) student evaluation of quality: and (uii) 
peer review of teaching processes. One should note that five of 
these are subjective measures. 

11. The question that besets most indicator projects is the following. 
Having collected data, what to do next? Indicators do not, in 
themselves, specify the action to take. Action requires 
interpretation of the meaning of indicators within a broader 
understanding of the context. Inevitably, there is a comparative 
dimension to indicator projects, even if it is not explicit. 
Comparisons are valuable - perhaps indispensable - in informing 
and guiding policy and action. Comparisons may be of self over 
time, in regard to targets, or of similar systems or organizations. 
Since most indicators are heavily dependent on social, political, 
and economic contexts, comparisons of institutions or of systems 
must be undertaken cautiously with knowledge of and respect for 
contextual differences. 

12. Once indicator information is available, there is a tendency for it 
to be used for purposes for which it was not designed. Within the 
more market-oriented systems, indicators at institutional level 
provide commercially sensitive information that is of obvious 
interest to prospective students. System-level indicators may, of 
course, attract little attention of this kind. The potential misuse of 
data can be reduced if protocols are established for data handling. 

13. Finally, the collection of good data is costly, and adequate 
resources need to be available and set aside for collection, 
analysis, and reporting. Generally speaking, institutional co- 
operation in providing data for system-level indicators is vital 
because it is feasible to collect certain information only at 
institutional level. 



3 The Australian Higher Education system uses the Course Experience Questionnaire 
(CEQ) (Ramsden, 1991) in an annual survey of graduates to measure the quality of teaching 
on the basis of a field of study. The instrument has scales measuring good teaching, 
feedback to students, student workload, generic skill development, and overall satisfaction. 
The measurement of teaching quality has been controversial and subject to some criticism. 
A federally funded project has recently recommended the incorporation of new scales into 
the CEQ to measure additional dimensions of the student ratperience (Mclnnis etcd., 2001). 
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4. CONSIDERATIONS IN THE SELECTION OF A STRATEGIC SET OF 
SYSTEM-LEVEL INDICATORS 

The bold objective of the “Strategic Indicators for Higher Education” project 
is to use indicators to influence higher education development and change 
in agreed upon directions. If all goes well, the project will contribute to a far- 
reaching chain of events; that is, the indicators that the project 
recommends will directly and indirectly: 

- reiterate the core elements in the vision articulated in the “World 
Declaration” and the “Priority Action Plan”; 

- support improved quantitative reporting of higher education; 

- focus attention on the interpretation of indicator data and improve 
understanding of the conclusions that can be drawn from data; 

- encourage and support informed policy and action towards the vision. 

Deciding upon an indicator framework is complex enough in any setting, 
but the selection of indicators appropriate at system-level involves 
particularly complex political, technical, and practical issues. 

Looking over the previous list of principles, some immediate conclusions 
can be drawn. The Priority Action Plan suggests system-level action for 
change and development across the three areas of inputs, processes, and 
outcomes, and suitable indicators might be chosen accordingly. Although 
there are many stakeholders in higher education, it is essential to treat 
policy-makers and administrators as the principal audience, for these 
people have the authority and the means to encourage and support effective 
data collection, analysis, and policy-making, at both system and 
institutional levels. Finally, the normative set of values for choosing the 
indicators is articulated by the vision for development proposed in the World 
Declaration and the Priority Action Plan. For the purposes of the project, 
these values are non-negotiable, even though some of the assumptions 
underlying them could be questioned. 

Ideally, the indicators should share the virtues listed below. It is highly 
unlikely, however, that individual indicators will meet each of these 
requirements. 

1. The indicators should be sufficiently provocative and relevant to 
encourage the commitment of resources to monitoring and action. 
They should therefore correspond with widely shared national and 
institutional values and priorities. 

2. They should be capable of definition that is meaningful across 
national and system differences without excessive caveats, and must 
be applicable in developed nations, developing nations, and less 
developed nations. 

3. They should measure system-level qualities rather than institutional- 
level qualities; however, they should be sufficiently relevant at 
institutional level. 
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4. They should be easy to interpret for all constituencies. 

5. They should be sufficiently detailed to be capable of revealing small 
developments over time. Tliey must also be stable over time to allow 
continuity of measurement. 



5. ISSUES RAISED AND DISCUSSED AT THE INVITATIONAL ROUNDTABLE 

In considering the possibilities for developing a preUminaiy set of 
recommendations for presentation to the Roundtable, three issues emerged. 
These are outlined below, together with the author’s personal views 
regarding each of them. 

L Are existing data sets capable of adequately representing the vision? 
All else being equal, it would be best if the recommended indicator set 
could be derived directly from readily available sources, or be at least 
extractable from existing data sets. However, Abercromby and 
Fielden’s (2000) survey of indicators in the context of the “World 
Declaration” and the “Priority Action Plan” reveals the absence of data 
in many important areas and tenuous links between goals and 
existing data in others. If the recommendations of the project were to 
be limited to available data, the project might fail in its main purpose 
- that is, to be strategic. It seems unavoidable that the project 
recommend indicators that would suggest that some, or all. Member 
States must collect new data. 

This task would be a highly strategic step for the project to take, 
one which corresponds to an objective of assisting Member States and 
institutions to improve the reporting of higher education through the 
development of data definitions and collection processes. While cost 
must clearly be considered, two or three yearly data collection cycles 
or spot surveys might be appropriate and would allow adequate time 
for developments to be measured. This area might be an appropriate 
one in which developed nations could assist less developed nations in 
data collection. 

it What about the aspects of the vision that are not quantifiable? The task 
would be more straightforward if all the goals expressed in the vision 
were quantifiable. However, core elements of the vision are not 
quantifiable in any acceptable way. One should consider, for example, 
the commitments to academic freedom and institutional autonomy. 
The conditions required for these freedoms are exceedingly difficult to 
quantify. Yet, omitting such values and goals from the framework on 
the grounds that they can only be assessed qualitatively would 
seriously damage the relevance of the project and weaken its 
relationship to the vision. It therefore might be necessary to consider 
at least a small number of qualitative indicators requiring subjective 
measurement and reporting. Depending on the circumstances, it 
might be feasible to use three- or five-point scales [e.g. 
low/ moderate /high or item-specific terminology). One should note 
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that the OECD (1998) reports the locus of educational decision- 
making by mode of decision-making using subjective survey data. 
at How can the project support the appropriate and effective interpretation 
of indicators? In addition to the specification of indicators, the project 
might benefit from considering the development of a support 
framework that includes recommendations for indicator interpretation 
within systems and institutions. Usually, it is helpful to suggest how 
an indicator might be interpreted and used. It would also be helpful to 
indicate how an indicator should not be used. In this regard, the 
implicit comparative dimension of the project requires further 
consideration. Arguably, an indicator is only useful if it allows 
development to be charted over time or benchmarking against 
comparable systems/ organizations. At the least, the project assumes 
comparison against itself over time. It is anticipated that Member 
States will monitor their development by looking for trends in their 
own indicator data. A potentially valuable additional outcome would 
be a framework for appropriate comparison with like partners. With 
appropriate support to qualitative investigation and analysis, 
comparison of this kind might identify particularly effective policies 
and actions. Any comparative work must of course acknowledge 
contextual differences. Expertise in comparative analysis would only 
be developed over a considerable period of time. 

6. SOME PRELIMINARY SUGGESTIONS 

A preliminary attempt to select a small set of system-level indicators follows. 
The approach taken is described below. 

The “Priority Action Plan” establishes well over sixty goals. As a first step 
in proposing a small and strategic set of indicators, it was necessary to 
distill the central values and priorities that relate to system-level action.^ 
This task involved some reduction and condensation of the numerous 
priorities outlined in the plan and some unavoidable losses of subtlety. In so 
doing, however, it became clear that the goals implied in the Priority Action 
Plan are rarefy discrete: they are often closely interrelated or overlapping. 
Furthermore, each goal may require multiple actions and some actions will 
affect a number of goals: the goal-action relationship is many-to-many. 
(Implying that the value of the indicators will derive from the set overall 
rather than from individual indicators in isolation.) 

This initial “pruning” exercise isolated four priority areas for system-level 
development in higher education, each with sub-categories, which reflect 
the areas in the Priority Action Plan for which indicators would be desirable. 



^ There is of course considerable overlap between system-level and institutional-level 
indicators. However, institutional-level indicators are largely a subset of system-level indicators. 
Consequently, not all system-level indicators are appropriate at institutional level; however, 
most, if not all, institutional-level indicators might be relevant at system level, or have close 
counterparts at system level. 
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Table 1 presents the four areas as a condensed framework for the Priority 
Action Plan. 

Table 1. Condensed framework for the UNESCO Priority Action Plan 



Indicators of an enabling policy and policy-making framework 

1 . 1 Accountable policy and decision-making framework: national and institutional 

1.2 Clear policies for higher education academic staff 

1.3 Promotion and development of research 

1.4 Conditions for freedom and autonomy (institutional, academic, student) 

Indicators of the commitment of resources 

2. 1 Fulfilled commitment of resources to higher education 

2.2 Increased co-operation among countries with regard to higher education and research, 
especially to reduce a widening gap between industrially developed and developing nations 

2.3 Use of new technologies 

Indicators of appropriate levels of participation, access, and retention 

3. 1 Expansion of access 

3.2 Equity of access 

3.3 Provision of student support 

Indicators of economic and social outcomes 

4. 1 Links between higher education, industry, and graduate employment 

4.2 Promotion of international mobility 

4.3 Catalytic effects on education systems overall and on local, regional, and national 

development 

Source: The author. 

Tables 2, 3, 4, and 5, on the pages that follow, present the sub-categories 
of each of these areas and list the elements of the “Priority Action Plan” to 
which each sub-category relates. 

The tables also include (very) tentative suggestions for indicators. 
Wherever there are obvious gaps, they are highlighted. The gaps occur for 
priorities that are strongly process-oriented and where the existence of 
processes and outcomes are difficult to quantify. No attempt has been made 
at this stage to offer precise data definitions. 

Table 2. Indicators of an enabling policy and of a policy-making framework 

Actions/Goals Possible indicators from which to choose 

Accountable policy and decision-making 
frameworks, national and institutional 
1(h) - involvement of all relevant stakeholders 
l(i) - enhancement women’s involvement in 
decision-making 

l(k)- involvement of students in [institutional] 
policy decisions 

6(g) - high quality, internal and external 
[institutional] evaluation 



Tl'iese actions /goals are highly subjective 
and not easily specified quantitatively 
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Actions/Goals 


Possible indicators from which to choose 


Clear policies for higher education teachers 
l(j )- clear policies concerning higher education 
teachers 

6 (d) - participation [of all academic staff) in 
teaching, research 


Academic salaries (adjusted using OECD 
Purchasing Power Parity data) 

Percent of staff time spent on R&D 
Percent of staff time spent on teaching 
Percent of academic staff holding doctorates 
(Opportunities for professional development, 
performance review?) 


Promotion and development of research 
1(b) - reinforcement of links between higher 
education and research 

1(e) - close links between higher education and 
research institutions 

6(i) - promotion and development of research [in] 
all higher education disciplines 


Expenditure on R&D in higher education 
overall and by discipline as a percentage of 
overall national expenditure on R&D 
Research higher degree students as a 
percentage of all students 
Percentage of staff time spent on R&D 
Expenditure (adjusted using OECD PPP data) 
on R&D per academic staff member 
Proportion of academic staff involved in 
research 

Proportion of higher education R&D financed 
by private enterprise 


Source: The author. 




Table 3. Indicators of the commitment of resources 


Actions/Goals 


Possible indicators from which to choose 


2. 1 Resource commitment to higher education 
1(g) - commitment of human, material and 
financial resources 


Higher education funding overall and by 
source, public or private, as percent of GDP 
Resources (total public and private) per full- 
time student equivalent (adjusted using 
OECD PPP data) 

Expenditure on teaching as a percentage of 
overall expenditure 


Increased co-operation between countries with 
regard to higher education and research, 
especially to reduce the widening gap between 
industrially developed and developing nations 
4 - increasing co-operation between all countries 
at all levels of economic development 
4 - reduce widening gaps between industrially 
developed and developing countries 


?? 


Use of new technologies 

8 - use of new technologies being generalized to 
the greatest extent possible to help higher 
education institutions 


Percent of higher education course time 
using information /communications 
technologies 



Source: The author. 
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Table 4. Indicators of appropriate levels of participation, access, and retention 

Actions/goals Possible indicators from which to choose 

3. 1 Expansion of access Students (domestic) overall and by 

2 - [where necessary] diversifying and expanding discipline, per 100,000 of the population 
access Percentage of students 25 years old and over 

1(d) - choice and flexibility of entry and exit points (commencing domestic students) 



1 (d) - lifelong learning 

3. 2 Equity of access 

1(a) - accessible to all on basis of merit 

1 (i) - consolidation of women’s participation 

3 - creation of gateways . . . especially for older 

students 



3.3 Provision of student support 
6(k) - fonns of student support, including 
measures to improve student living conditions 
Source: The author. 



Basis of admission 

Ratio of higher education participation to the 
proportion of students of lower socio- 
economic background to proportion of 
people of lower socio-economic background 
in the population 

Percentage of first-generation higher 
education students, among commencing 
students 

Ratio of men to women, among commencing 
domestic students 

Women enrolled in research higher degree 
programmes as a percentage of all students 
enrolled in research higher degree 
programmes 

Percentage of students enrolled part-time 

First-year retention rates 
Student (all) /teaching staff ratio 
Student (all) /all staff ratio 



Table 5. Indicators of economic and social outcomes 



Actions/Goals 



Possible indicators from which to choose 



Links between higher education, industry, and 
graduate employment 

7 - closer links between higher education and 
the world of work 

Promotion of international mobility 
1 (m) ~ facilitation of international and national 
mobility of staff and students 



Catalytic effects, on education systems overall 

and on local, regional, and national development 

1 (c) - catalyst for entire education system 

1(0 - contribution to local, regional, and national 

development 

Source: The author. 



Graduate employment rates: two-years-out, 
overall, and by discipline 

International students as a percentage of all 
students 

Percentage of academic staff members with 
highest qualification awarded overseas 



Total expenditure on education overall and by 
level as a percentage of GDP 
Participation rates in education by level 



Table 6 comments on the possible utility of certain quantitative 
indicators. Table 7 lists the approximate data requirements for the 
indicators listed. 
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Some of the possible indicators are simple proxies for the action/goal in 
question. Admittedly some are at a considerable conceptual distance from 
the outcome they might represent, raising concerns about face validity. 

Table 6. Notes regarding the possible quantitative indieators listed 



- The participation rates of socio-economic subgroups are notoriously difficult to measure. The 
usual socio-economic data are income, employment type, and level of education. For 

purposes of higher education access, parental educational level is a useful measure of “first 
generation” access. 

Data on commencing students is more sensitive in detecting the effects of changes and 
developments than data on all students. 

Discipline and level [e.g., participation in research-oriented higher degrees) best gauge the 
participation of women. 

- Research consistently shows that the first year of study is a critical time for retention. First 
year retention rates serve as a proxy for the range and quality of student support seiwices. 

lo measure the promotion and development of research, expenditure on R&D in higher 
education is best presented by discipline as well as an overall aggregate. 

^^^^clj-oriented higher degree students as a percentage of all students is a good indicator of 
the level of R&D; however, disciplrnaiy variations need to be considered. 

To gain an accurate picture of graduate outcomes (employment rates and destinations), 
measurement needs to take place at least one year, possibly two years after graduation. The 
reporting of outcomes by broad discipline is valuable. 

- Along with age, basis of admission is a good indicator of access/lifelong learning. 

Crude currency data (e.g., academic salaries) should be adjusted using OECD Purchasing 
Power Parity data. ® 



Some consideration must be given to whether or not the indicators 
should attempt to measure outcomes or the existence of actions to produce 
outcomes. An emphasis on direct measurement of outcomes is not always 
possible but perhaps highly preferable. 

There are two reasons for such emphasis. First, outcomes are more likely 
to be quantifiable. Second, the relationship between actions and outcomes 
can be obscure, and cause-effect relationships may vaiy across social and 
cultural contexts. In the end, what is most important perhaps is evidence of 
development in the preferred direction, where necessary, rather than how 
the development is achieved. 

Electronic technology will continue to have far-reaching effects in higher 
education. There are obviously sharp differences between usage and 
purposes in developed and developing nations; however, the use of 
educational technology is an indicator of modernization and (possibly) of the 
effectiveness of access and of quality of learning. The suggested “percent of 
course time using technology” is clumsy, but would better reflect the 
penetration of information technology into the curriculum than a simple 
volume-of-hardware indicator. 

The internationalization of higher education proceeds apace, and this 
trend, coupled with the electronic deliveiy of courses, will significantly affect 
many indicators in the future. Notable tn the context of the likely impact on 
the utility of system-level indicators are international course franchising 
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and cross-national enrollments in on-line delivery. The United States is 
already a major provider of full on-line courses. Various international 
consortia are rapidly forming. These are indications that these 
developments will grow for at least the next decade. The new patterns of 
international enrollment are not yet weU understood, yet they will seriously 
confuse the interpretation of certain system- level indicators in the future. 

Table 7. Approximate data requirements for the indicators listed 



Domestic students (all and commencing) 

Number overall 
by sex 
by age 

by SES (parental education level?) 
by field of study 

by enrollment type (full-time /part-time) 
by level of study (undergraduate/postgraduate) 
by type of study (research /coursework) 

Basis of admission 
International students 
Number overall 
by field of study 

by level of study (undergraduate /postgraduate) 
by type of study (research/coursework) 

First year retention rate 
Staff 

Number overall 
by sex 
by field 

by location of highest academic qualification 

percentage of time allocated to teaching, research, and service 

Academic salaries 

Course time using information /communications technologies 

Graduate employment rates two-years-out 

Resources 

Revenue by source (public and private) 

Expenditure (on teaching and on R&D) 

R&D financed by private enteiprises 
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